Some test harnesses allow to pass a --binary argument to give the path of a firefox that doesn't come from a local build. All harnesses should have such an option. For instance, I wanted to run locally a crashtest with a try build... (under rr), but can't out of the box.
In case of some this applies to Marionette and Firefox UI tests right now. It's done like that: https://dxr.mozilla.org/mozilla-central/source/testing/marionette/mach_commands.py#91-92 Greg, which mach file would we have to modify to make it globally available for test commands? Is that possible to do at a single location or would all the test harnesses need an update?
I'd put common code in python/mozbuild/mozbuild/base.py. That's already where we define many command conditions. Also, if you do this, please make a custom decorator for adding common arguments to test commands. Right now, test command arguments vary in subtle ways. This hinders usability by introducing inconsistency.
For a concrete example of inconsistency between commands, see bug 1388117.
So as it looks like the --binary option as used by Marionette's mach command isn't defined there, but it's just forwarded to Marionette itself. This means if we want to have the same cli argument, each of our test harnesses would have to be updated to allow the --binary argument. Ted, what do you think? In case of Marionette tests we clearly have a huge benefit for that all the last months.
I think we should support this in all test harnesses. Whatever you have to do behind the scenes to make that work is fine.