Closed Bug 1696251 Opened 3 years ago Closed 3 years ago

Allow Mach commands as standalone functions

Categories

(Firefox Build System :: Mach Core, enhancement)

enhancement

Tracking

(firefox94 fixed)

RESOLVED FIXED
94 Branch
Tracking Status
firefox94 --- fixed

People

(Reporter: mhentges, Assigned: alex.lopez.zorzano)

References

Details

Attachments

(16 files, 3 obsolete files)

48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review

Currently, to create a mach command, you must do:

@CommandProvider
class CustomCommandClass(MachCommandBase):
    @Command(...)
    @CommandArgument(...)
    @CommandArgument(...)
    def my_command(self, argument1, argument2):
        # Command arguments are arguments to this function,
        # context information is stored as member variables of MachCommandBzse
        print("That was a lot of ceremony!")

The downsides of this are:

  • There's more ceremony in the code (what is this, Java?)
  • Adding an __init__(...) requires that you bubble parameters up to MachCommandBase.
  • It requires the creation of MachCommandBase to test mach commands - it's hard to "mock out" parent classes.

Instead, we should allow defining commands as:

@Command(...)
@CommandArgument(...)
@CommandArgument(...)
def my_command(mach_context, argument1, argument2):
    # What used to be stored in `self` is now in `mach_context`.
    # Arguments are still provided as function parameters
    print("This was so convenient, mach is incredible software")

EDIT: See comment 13 for more context.

Blocks: 1695312

This gives the Command and SubCommand decorators the capability to register
commands specified as standalone functions, while retaining the compatibility
with classes as before.

Assignee: nobody → alex.lopez.zorzano
Status: NEW → ASSIGNED

In preparation for future changes in how these decorators work,
add a few basic tests.

Depends on: 1697042
Attachment #9206960 - Attachment is obsolete: true

As a step to remove the need to use @CommandProvider, we are moving
the metrics_path parameter to @Command.

Depends on D107836

No longer blocks: 1695312
Blocks: 1388894
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3e834be2046b
Add tests to mach command decorators. r=mhentges
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → 88 Branch

Still more work to do yet :)

Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Status: REOPENED → ASSIGNED
Keywords: leave-open
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a8dcd8e64a85
Move metrics_path parameter from @CommandProvider to @Command/@SubCommand . r=mhentges

As an intermediate step to allow mach commands as standalone functions, the MachCommandBase
subclass instance that currently corresponds to self has to be made available as a separate
argument (named command_context).

As a step towards moving mach commands outside of classes, this converts all
properties into methods so that they can later become top-level helper functions.

Depends on D109650

Some context for teams tuning into this bug because a Mach command they maintain is being updated by Alex:

All of our Mach commands right now are stored as functions in classes, but the class abstraction is unnecessary - all the methods of such a CommandProvider can be happily moved up to the class-level, simplifying the code and making future mach commands more easy to write.

However, to achieve this, we need to make some tweaks:

  • Modules can't have @property functions (of course), so we're turning properties into regular helper functions.
  • Modules can't (well, shouldn't) have initialization logic, so pieces from __init__() are being moved into helper functions.
  • There will no longer be a self, so we're replacing that with a command_context variable that has the same goodies.
  • Then, at the end, we can remove the top-level class and shuffle everything up a level.

We're attempting to do these changes in a way that's both easy-to-review and keeps the entire tree consistent.
So, if you're surprised by some changes happening to your local Mach command, then they can probably be explained by ^ :)

Another step towards avoiding the need for classes in mach commands;
here we are removing constructors either by changing them into helpers
or by simple refactorings.

Depends on D112196

Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d4a5d8567977
Pass MachCommandBase object as first argument for Mach Commands. r=mhentges,remote-protocol-reviewers,marionette-reviewers,webdriver-reviewers,perftest-reviewers

Hmm, looks like ./mach try auto didn't know all the affected commands.
Let's get those failed tasks in that "Push with failures" resolved, then we can land again.

Flags: needinfo?(alex.lopez.zorzano)
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e169193b7423
Pass MachCommandBase object as first argument for Mach Commands. r=mhentges,remote-protocol-reviewers,marionette-reviewers,webdriver-reviewers,perftest-reviewers
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1c84c9a34575
Turn all properties in MachCommandBase subclasses into methods. r=mhentges,perftest-reviewers,sparky
Flags: needinfo?(alex.lopez.zorzano)
Attachment #9216518 - Attachment description: Bug 1696251 - Refactor constructors in MachCommandBase subclasses to remove them. r=mhentges → Bug 1696251 - WIP Refactor constructors in MachCommandBase subclasses to remove them. r=mhentges
Attachment #9216518 - Attachment description: Bug 1696251 - WIP Refactor constructors in MachCommandBase subclasses to remove them. r=mhentges → WIP: Bug 1696251 - Refactor constructors in MachCommandBase subclasses to remove them. r=mhentges
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e3f73f78f1c7
Pass MachCommandBase object as first argument for Mach Commands. r=mhentges,remote-protocol-reviewers,marionette-reviewers,webdriver-reviewers,perftest-reviewers
Pushed by thunderbird@calypsoblue.org:
https://hg.mozilla.org/integration/autoland/rev/e949c39b93e6
Fix "mach taskgraph full" command. r=mhentges
Attachment #9216518 - Attachment description: WIP: Bug 1696251 - Refactor constructors in MachCommandBase subclasses to remove them. r=mhentges → Bug 1696251 - Refactor constructors in MachCommandBase subclasses to remove them. r=mhentges
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/eddb5c40cd92
Turn all properties in MachCommandBase subclasses into methods. r=mhentges,perftest-reviewers,sparky
Regressions: 1713060
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/4b889750da57
Refactor constructors in MachCommandBase subclasses to remove them. r=mhentges,remote-protocol-reviewers
Flags: needinfo?(alex.lopez.zorzano)
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/134e5702f160
Refactor constructors in MachCommandBase subclasses to remove them. r=mhentges,remote-protocol-reviewers

This step removes all the dependencies of mach commands to
having a MachCommandBase as the self by using the command_context
argument instead. This also removes any remaining statefulness from those
classes that implement mach commands, ultimately making it easier to move
existing commands out of classes in a follow-up.

Attachment #9213850 - Attachment is obsolete: true
Attachment #9213489 - Attachment is obsolete: true
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e1921c5112d8
Replace self with command_context where possible in existing mach commands. r=mhentges,webdriver-reviewers,perftest-reviewers,whimboo
Flags: needinfo?(alex.lopez.zorzano)
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/63279eccfdfc
Replace self with command_context where possible in existing mach commands. r=mhentges,webdriver-reviewers,perftest-reviewers,whimboo

When transitioning from using self to using command_context in mach
commands, this specific mistake went undetected (no bustage forced a backout,
either). This commit fixes it.

Attachment #9233021 - Attachment description: WIP: Bug 1696251 - Fix error in static analysis mach_commands.py → Bug 1696251 - Fix error in static analysis mach_commands.py. r=mhentges
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9075b050f763
Fix error in static analysis mach_commands.py. r=mhentges

This removes the @CommandProvider decorator and the need to implement
mach commands inside subclasses of MachCommandBase, and moves all
existing commands out from classes to module level functions.

Depends on: 1723599
No longer depends on: 1723599
Regressions: 1723599
Target Milestone: 88 Branch → ---

The patch that made the replacement across all commands
missed this particular file.

Attachment #9234233 - Attachment description: Bug 1696251: Allow mach commands as stand-alone functions and adapt existing commands. r=mhentges → WIP: Bug 1696251: Allow mach commands as stand-alone functions and adapt existing commands. r=mhentges
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ecfc828f27fc
Replace usages of self with command_context in compare-locales mach commands. r=mhentges

This is to keep up with changes happening in commands while still transitioning
to commands without classes.

Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/cacb7e90783b
Replace usages of self with command_context in mozbuild msix mach commands. r=mhentges

The purpose of this is to remove as many docstrings from CommandProvider
classes to make the step of moving commands out of classes simpler.
Where possible, the docstring has been moved to or merged with the function.

The affected functions would cause an UnboundLocalError if they are moved
to module-level, because using an assignment inside a function makes a
variable local in Python, making it illegal to have a local variable name
be the same as a function name that is called within the same function.
The adopted solution is to rename those functions in preparation for the
future de-classing.

Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3602cbaf8893
Change method names in mach commands to avoid scoping issues when de-classing. r=mhentges
Attachment #9234233 - Attachment description: WIP: Bug 1696251: Allow mach commands as stand-alone functions and adapt existing commands. r=mhentges → Bug 1696251: Allow mach commands as stand-alone functions and adapt existing commands. r=mhentges
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/5f5b612878f3
Allow mach commands as stand-alone functions and adapt existing commands. r=mhentges,webdriver-reviewers,perftest-reviewers,sparky,whimboo
Regressions: 1731396
Backout by nerli@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e28c911d36db
Backed out changeset 5f5b612878f3 for causing multiple bustages
Flags: needinfo?(mhentges)
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f25fa6972142
Rename setup to avoid name conflict when declassing mach commands. r=mhentges,perftest-reviewers,AlexandruIonescu
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/510dd46a9de7
Allow mach commands as stand-alone functions and adapt existing commands. r=mhentges,webdriver-reviewers,perftest-reviewers,sparky,whimboo
Regressions: 1731703

Backed out for causing bustages

[task 2021-09-21T01:44:34.588Z] 01:44:34     INFO -  01:44:34     INFO -  List of locales does not include default locale "en-US": ['an', 'ar', 'ast', 'az', 'be', 'bg', 'bn', 'br', 'bs', 'ca', 'cak', 'cs', 'cy', 'da', 'de', 'dsb', 'el', 'en-CA', 'en-GB', 'eo', 'es-AR', 'es-CL', 'es-ES', 'es-MX', 'et', 'eu', 'fa', 'ff', 'fi', 'fr', 'fy-NL', 'ga-IE', 'gd', 'gl', 'gn', 'gu-IN', 'he', 'hi-IN', 'hr', 'hsb', 'hu', 'hy-AM', 'id', 'is', 'it', 'ja', 'ka', 'kab', 'kk', 'kn', 'ko', 'lij', 'lo', 'lt', 'lv', 'ml', 'mr', 'ms', 'my', 'nb-NO', 'ne-NP', 'nl', 'nn-NO', 'oc', 'pa-IN', 'pl', 'pt-BR', 'pt-PT', 'rm', 'ro', 'ru', 'sk', 'sl', 'son', 'sq', 'sr', 'sv-SE', 'ta', 'te', 'th', 'tr', 'trs', 'uk', 'ur', 'uz', 'vi', 'wo', 'xh', 'zam', 'zh-CN', 'zh-TW']; adding "en-US"
[task 2021-09-21T01:44:34.590Z] 01:44:34     INFO -  01:44:34     INFO -  Error running mach:
[task 2021-09-21T01:44:34.591Z] 01:44:34     INFO -  01:44:34     INFO -      ['--log-no-times', 'package-multi-locale', '--locales', 'an', 'ar', 'ast', 'az', 'be', 'bg', 'bn', 'br', 'bs', 'ca', 'cak', 'cs', 'cy', 'da', 'de', 'dsb', 'el', 'en-CA', 'en-GB', 'eo', 'es-AR', 'es-CL', 'es-ES', 'es-MX', 'et', 'eu', 'fa', 'ff', 'fi', 'fr', 'fy-NL', 'ga-IE', 'gd', 'gl', 'gn', 'gu-IN', 'he', 'hi-IN', 'hr', 'hsb', 'hu', 'hy-AM', 'id', 'is', 'it', 'ja', 'ka', 'kab', 'kk', 'kn', 'ko', 'lij', 'lo', 'lt', 'lv', 'ml', 'mr', 'ms', 'my', 'nb-NO', 'ne-NP', 'nl', 'nn-NO', 'oc', 'pa-IN', 'pl', 'pt-BR', 'pt-PT', 'rm', 'ro', 'ru', 'sk', 'sl', 'son', 'sq', 'sr', 'sv-SE', 'ta', 'te', 'th', 'tr', 'trs', 'uk', 'ur', 'uz', 'vi', 'wo', 'xh', 'zam', 'zh-CN', 'zh-TW']
[task 2021-09-21T01:44:34.591Z] 01:44:34     INFO -  01:44:34     INFO -  The error occurred in the implementation of the invoked mach command.
[task 2021-09-21T01:44:34.591Z] 01:44:34     INFO -  01:44:34     INFO -  This should never occur and is likely a bug in the implementation of that
[task 2021-09-21T01:44:34.591Z] 01:44:34     INFO -  01:44:34     INFO -  command.
[task 2021-09-21T01:44:34.591Z] 01:44:34     INFO -  01:44:34     INFO -  You can invoke |./mach busted| to check if this issue is already on file. If it
[task 2021-09-21T01:44:34.591Z] 01:44:34     INFO -  01:44:34     INFO -  isn't, please use |./mach busted file package-multi-locale| to report it. If |./mach busted| is
[task 2021-09-21T01:44:34.591Z] 01:44:34     INFO -  01:44:34     INFO -  misbehaving, you can also inspect the dependencies of bug 1543241.
[task 2021-09-21T01:44:34.591Z] 01:44:34     INFO -  01:44:34     INFO -  If filing a bug, please include the full output of mach, including this error
[task 2021-09-21T01:44:34.591Z] 01:44:34     INFO -  01:44:34     INFO -  message.
[task 2021-09-21T01:44:34.591Z] 01:44:34     INFO -  01:44:34     INFO -  The details of the failure are as follows:
[task 2021-09-21T01:44:34.591Z] 01:44:34     INFO -  01:44:34     INFO -  AttributeError: 'list' object has no attribute '_get_state_filename'
[task 2021-09-21T01:44:34.592Z] 01:44:34     INFO -  01:44:34     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/mach_commands.py", line 2281, in package_l10n
[task 2021-09-21T01:44:34.592Z] 01:44:34     INFO -  01:44:34     INFO -      locales = list(sorted(locales))
[task 2021-09-21T01:44:34.592Z] 01:44:34     INFO -  01:44:34     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/mach_commands.py", line 476, in list
[task 2021-09-21T01:44:34.592Z] 01:44:34     INFO -  01:44:34     INFO -      database = get_warnings_database(command_context)
[task 2021-09-21T01:44:34.592Z] 01:44:34     INFO -  01:44:34     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/mach_commands.py", line 407, in get_warnings_database
[task 2021-09-21T01:44:34.592Z] 01:44:34     INFO -  01:44:34     INFO -      path = database_path(command_context)
[task 2021-09-21T01:44:34.592Z] 01:44:34     INFO -  01:44:34     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/mach_commands.py", line 401, in database_path
[task 2021-09-21T01:44:34.592Z] 01:44:34     INFO -  01:44:34     INFO -      return command_context._get_state_filename("warnings.json")
[task 2021-09-21T01:44:34.635Z] 01:44:34     INFO -  01:44:34    ERROR - Return code: 1
[task 2021-09-21T01:44:34.635Z] 01:44:34     INFO -  01:44:34    FATAL - 'mach package-multi-locale --locales an ar ast az be bg bn br bs ca cak cs cy da de dsb el en-CA en-GB eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja ka kab kk kn ko lij lo lt lv ml mr ms my nb-NO ne-NP nl nn-NO oc pa-IN pl pt-BR pt-PT rm ro ru sk sl son sq sr sv-SE ta te th tr trs uk ur uz vi wo xh zam zh-CN zh-TW' did not run successfully. Please check log for errors.
[task 2021-09-21T01:44:34.635Z] 01:44:34     INFO -  01:44:34    FATAL - Running post_fatal callback...
[task 2021-09-21T01:44:34.635Z] 01:44:34     INFO -  01:44:34    FATAL - Exiting -1
Flags: needinfo?(alex.lopez.zorzano)

Please make sure to address bug 1731703 before relanding (which was also the error in the second log in comment 54).

Attachment #9242339 - Attachment description: Bug 1696251 - Rename list to list_warnings to avoid shadowing builtin list when de-classing. r=mhentges → WIP: Bug 1696251 - Rename methods that would shadow builtin functions when de-classing. r=mhentges
Attachment #9234233 - Attachment description: Bug 1696251: Allow mach commands as stand-alone functions and adapt existing commands. r=mhentges → WIP: Bug 1696251: Allow mach commands as stand-alone functions and adapt existing commands. r=mhentges
Attachment #9234233 - Attachment description: WIP: Bug 1696251: Allow mach commands as stand-alone functions and adapt existing commands. r=mhentges → Bug 1696251: Allow mach commands as stand-alone functions and adapt existing commands. r=mhentges
Attachment #9242339 - Attachment description: WIP: Bug 1696251 - Rename methods that would shadow builtin functions when de-classing. r=mhentges → Bug 1696251 - Rename methods that would shadow builtin functions when de-classing. r=mhentges
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/803ef9f68ea8
Rename methods that would shadow builtin functions when de-classing. r=mhentges
https://hg.mozilla.org/integration/autoland/rev/53b1fa0faa6d
Allow mach commands as stand-alone functions and adapt existing commands. r=mhentges,webdriver-reviewers,perftest-reviewers,sparky,whimboo

Fantastic work Alex! The widespread refactor is complete 😅

I'm going to broadcast this a bit later today and touch on its main benefits:

  • Less boilerplate/complexity when defining Mach commands
  • Mach commands are far easier to test, because you no longer need to do a parent-class monkeypatch to instrument your command without having unrelated MozbuildObject/MachCommandBase work going on (parsing mozconfig, etc).
Status: ASSIGNED → RESOLVED
Closed: 3 years ago3 years ago
Flags: needinfo?(alex.lopez.zorzano)
Resolution: --- → FIXED
Regressions: 1732255
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Backed out for breaking the static-analysis integration: https://hg.mozilla.org/mozilla-central/rev/c38d34be7c3f757510f037fef5a78ea08af32980

Flags: needinfo?(alex.lopez.zorzano)

In this patch, we differentiate methods that actually come from the Build
class from those that actually come from the parent MachCommandBase class,
which will have to be called on the command_context object. Also, we rename
builder to build_commands in preparation for those methods becoming top-level
functions of the module of the same name.

Flags: needinfo?(alex.lopez.zorzano)
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/5c95fce39b9c
Adjust usages of mozbuild/build_commands.py's Build to simplify future declassing. r=mhentges
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/94ababc37c90
Allow mach commands as stand-alone functions and adapt existing commands. r=mhentges,webdriver-reviewers,perftest-reviewers,sparky,whimboo
Status: REOPENED → RESOLVED
Closed: 3 years ago3 years ago
Resolution: --- → FIXED
Target Milestone: --- → 94 Branch
Regressions: 1733061
Regressions: 1733312
Blocks: 1733668
Regressions: 1736950
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: