1879187 - Safari benchmarking tasks unnecessarily depend on having a Firefox build

Status: NEW

(Firefox Build System :: Task Configuration, enhancement)

Description

[Jeff Muizelaar [:jrmuizel]]

e.g. Raptor (browsertime) Benchmark tests on Safari on speedometer depends on build-macosx64-shippable/opt

It seems like we could run these without needing to wait for the corresponding Firefox build.

Comment 1

[Jeff Muizelaar [:jrmuizel]]

ahal, do you know how this would be accomplished?

Comment 2

[Andrew Halberstadt [:ahal]]

Those tasks are part of the test kind, which have a lot of assumptions that there is build to test. This is sort of the fundamental feature of that kind.

So I'd recommend either creating a new kind or re-using the source-test kind (where tasks don't necessarily depend on a build). The perftest team maintains these tasks, so I'll redirect the needinfo over to :sparky.

Comment 3

[Greg Mierzwinski [:sparky]]

We depend on the mozharness artifacts that the builds produce so it's not easy for us to switch raptor away from these builds (the same is true for our chrome tests). Ideally, we would move all of these tests to mozperftest which doesn't depend on mozharness, and allows us to run tests without the build prerequisite but that's a big task and it would take quite a while to do.

That said, one thing we could potentially do is some hack with the taskcluster transforms to get them to depend on a linux opt build instead of the mac builds. I ran a test here just to see if it would make it past the decision task and it's looking promising: https://treeherder.mozilla.org/jobs?repo=try&revision=ea98a986eb60d82143a86792cd1a83a366c69de3

:jrmuizel, if you want, we could try implementing that approach? This could also be extended to our chrome tests if it works out.

Comment 4

[Jeff Muizelaar [:jrmuizel]]

If it's not easy to fix, it might be ok to just live with this for now and aspire to do something about it in the future.