1798974 - [meta] Modify how benchmarks are run

Status: NEW

(Testing :: Raptor, task, P3)

Description

[Greg Mierzwinski [:sparky]]

Currently, the method we use for running benchmarks is quite inflexible. It requires us to make releases in the perf-automation repo: https://github.com/mozilla/perf-automation

It suffices to say that the process is convoluted for developers to use if they want to easily test updates to their benchmarks (such as matrix-react, or the twitch benchmark). On top of that, we can't get these built automatically in CI.

This bug is for two things:
(1) Modifying how the benchmarks are pulled in from repositories, and
(2) Adding the ability to run a build task for the tools (similar to what we do for browsertime).

In CI, we are almost ready for allowing non perf-automation benchmarks to be used but locally, we need to fix up mach_commands to let it pull from other repos whose location may be defined in the INI files.