1821708 - Add a Mann-Whitney U-test performance change detector to mozperftest as a new tool

Status: RESOLVED FIXED

(Testing :: mozperftest, task, P1)

Description

[Greg Mierzwinski [:sparky]]

This bug is for adding a performance change detector to mozperftest as a new tool. This includes a local tool, and a tool for CI.

Comment 1

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1821708 - Add performance change detector to mach perftest-tools. r?#perftest

This patch adds a new tool that runs a comparison between two, or more revisions to detect performance changes. Some changes are made to accommodate this new tool alongside the side-by-side tool. The tests for the detector coding are found in another patch in the series. A mozperftest-tools update to 0.2.5 is required for this change.

There is also a CI task that is added in this patch. It's setup in the mach try perf patch in this series, which also has more information.

Depends on D172281

Comment 2

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1821708 - Add tests for the change-detector.r?#perftest

This patch adds tests for the new change-detector tool.

Depends on D172282

Comment 3

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1821708 - Add change-detector to mach try perf. r?#perftest

This patch adds a method to inject the perftest change-detector task into mach try perf try runs. To do this, we need to flip the ordering of the try runs to do the base revision first, and the one with the changes (new) afterwards. This allows us to link a change-detector task in the new try run, with the base try run by passing a revision through the try-task-config.

The task can be triggered using --detect-changes. Once a user pushes to try, and after all of the performance tests finish in that push, the change-detector will start running, to generate a comparison. An artifact is produced with the results (revisions, and metrics regressed), and an exception is raise when a performance change is detected even if it's an improvement.

Depends on D172283

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c582db0ea6e7
Add performance change detector to mach perftest-tools. r=perftest-reviewers,kshampur,AlexandruIonescu
https://hg.mozilla.org/integration/autoland/rev/5b91abd32a17
Add tests for the change-detector.r=perftest-reviewers,afinder
https://hg.mozilla.org/integration/autoland/rev/7774728d64f1
Add change-detector to mach try perf. r=perftest-reviewers,afinder

Comment 5

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/c582db0ea6e7
https://hg.mozilla.org/mozilla-central/rev/5b91abd32a17
https://hg.mozilla.org/mozilla-central/rev/7774728d64f1