1907926 - Build a tool/command that can run all the tests that alerted locally

Status: RESOLVED FIXED

(Testing :: Performance, task, P2)

Description

[Greg Mierzwinski [:sparky]]

It would be really useful to developers to have a single command that they could run locally to run all the tests that alerted in an alert summary. This would be something similar to ./mach try perf --alert <NUM>. Maybe we could use mozperftest for it.

Comment 1

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1907926 - Add an local alert test run layer to mozperftest. r?#perftest

This patch adds a new layer to mozperftest that allows people to run all the tests that produced an alert locally.

It does this by querying Treeherder for the alert information, and determining the tests to run from there. A --alert-exact option is added to enable users to run the exact same command as was run in CI as well. This is done by querying the task information of the alerting task, and parsing the options from it in a framework-specific manner.

All harnesses have option parsing implemented for them, however some of them, such as AWSY are more manual due to how the argument parser is implemented in that harness.

Depends on D219964

Comment 2

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1907926 - Add tests for the alert layer. r?#perftest

This patch adds tests, and mock test data for the alert layer.

Depends on D219965

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b37a84f28d80
Add an local alert test run layer to mozperftest. r=perftest-reviewers,aglavic
https://hg.mozilla.org/integration/autoland/rev/4be97f09160b
Add tests for the alert layer. r=perftest-reviewers,aglavic

Comment 4

[Noemi Erli[:noemi_erli]]

Backed out 3 changesets (Bug 1907926, Bug 1914607) for breaking unit tests
Backout link: https://hg.mozilla.org/integration/autoland/rev/acdbfef291d266d071a10acf6dccb4a115742f8c

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6c3f81917e34
Add an local alert test run layer to mozperftest. r=perftest-reviewers,aglavic DONTBUILD
https://hg.mozilla.org/integration/autoland/rev/6147c7929945
Add tests for the alert layer. r=perftest-reviewers,aglavic DONTBUILD

Comment 6

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/6c3f81917e34
https://hg.mozilla.org/mozilla-central/rev/6147c7929945