1808747 - Add a way to associate a python support file to a test

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement, P2)

Description

[Greg Mierzwinski [:sparky]]

This bug is for adding a way to associate a python file to a test.

With a python file, we'll be able to greatly expand how much a test can customize it's setup, and parse it's results. This will make it simpler to handle all the various options we have for custom tests as well, see the grandprix test for instance: https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/custom/browsertime-grandprix.ini#25-28

One issue we have is that Raptor results parsing is intermingled between results.py and output.py. This results in a lot of issues when it comes to parsing results and it forces us on the PerfTools team to constantly modify these files to add tests. This is because the files are very rigid in that it's very error-prone to customize your perfherder-data. With python files, we'll be able to skip certain areas of those files with custom user-defined behaviour making it easier for developers to add tests themselves.

Comment 1

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1808747 - Add python test support class scripts to raptor. r?#perftest

This patch adds python support classes for tests. With them, we'll be able to customize the results, and output parsing better, and with less impact on other existing tests. The first method available allows a test author to modify the command before the test runs. A sample test is also added to show how it all works. No CI tests will be added for it.

Comment 2

[BugBot [:suhaib / :marco]]

There's a r+ patch which didn't land and no activity in this bug for 2 weeks.
:sparky, could you have a look please?
If you still have some work to do, you can add an action "Plan Changes" in Phabricator.
For more information, please visit auto_nag documentation.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9ee83fa2a0ed
Add python test support class scripts to raptor. r=perftest-reviewers,afinder

Comment 4

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/9ee83fa2a0ed