1644984 - perftests.yml to control CI runs

Status: NEW

(Testing :: mozperftest, enhancement, P2)

Description

[Tarek Ziadé (:tarek)]

we're going to add perftests in various location in our tree, and a dry call of

./mach perftest

will collect them and run them.

for this to work we need each test to provide its perftest arguments and if it should be running daily in the CI (and one what plaftorms)

We're adding the ability to run a python test instead of a JS test, which is great and we can do some of this there. but we'll still need to get the CI info.

This info should not be tied to the test itself.

What I propose is a yml file that can be declared in moz.build, that provides all the required information like how we would do in kind.yml files. things like:

• link to directories containing perftests for running batches
• list of tests that should be hidden from the list (so they are called only explicitly)
• list of tests that should be running in the CI and for each one, what platforms
• for each platform, a list of default arguments that we pass to perftest

mach perftest should be able to run against that yml file locally and in the CI.

This will also be the base for a fuzzy like UI where we would let the use pick some tests

Comment 1

[Tarek Ziadé (:tarek)]

I think the idea is roughly to adopt the same kind.yml-file like to build the test suite, for both local and CI runs

in the CI we can have a transformer that grabs them and create the jobs on the fly

Comment 2

[Greg Mierzwinski [:sparky]]

Yes to this! We'll also need something like this to specify settings on the metrics obtained (like whether it should alert or not).

Comment 3

[Andrew Creskey [:acreskey]]

I'm still a just trying to figure out the basics of taskcluster yml files.
But I like these ideas:

mach perftest should be able to run against that yml file locally and in the CI.

Being able to run the same config in both is great.

This will also be the base for a fuzzy like UI where we would let the use pick some tests
And this, yes, please.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

we typically add some variables on moz.build to get a list of manifests:
https://searchfox.org/mozilla-central/search?q=MANIFESTS&path=moz.build&case=true&regexp=false

Comment 5

[Tarek Ziadé (:tarek)]

yeah, what I am proposing it to have a variable that declares those yml files:

PERFTESTS = ["tests/perftests.yml"]

and have the yml file replace the usual ini file, with more features as described . having them described in ini files but it sounds more complex in a flat structure