1772015 - [perfdocs] Add more documentation about metrics gathered through Raptor

Status: RESOLVED FIXED

(Testing :: Raptor, task, P2)

Description

[Greg Mierzwinski [:sparky]]

This bug is for adding a bit more documentation about how our metrics are collected, and what they are to our documentation. It might also be useful to update our information about how we can do regression hunting because we have a lot of new tools e.g. mozperftest-tools, new browsertime videos with overlaid metrics, etc..

Comment 1

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1772015 - [perfdocs] Add metric documentation, and validation to Raptor. r?#perftest

This patch adds the ability to document metrics, and have them validated against metrics that are being measured by the test. The definitions can be done in any "level" of the configuration (test, suite, global), however there can't be any duplicate definitions for metrics (including their aliases).

Warnings are output for the following:

• Duplicate metrics definitions.
• Undocumented metrics that are measured (alerting for Raptor).
• Documented, but unmeasured metrics.

Changes to the tests are to make them work with the changes done in this patch.

Comment 2

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1772015 - [perfdocs] Add metric documentation, and validation to Raptor. r?#perftest

This patch adds the ability to document metrics, and have them validated against metrics that are being measured by the test. The definitions can be done in any "level" of the configuration (test, suite, global), however there can't be any duplicate definitions for metrics (including their aliases).

Warnings are output for the following:

• Duplicate metrics definitions.
• Undocumented metrics that are measured (alerting for Raptor).
• Documented, but unmeasured metrics.

Changes to the tests are to make them work with the changes done in this patch.

Comment 3

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1772015 - [perfdocs] Add metric documentation, and validation to Raptor. r?#perftest

This patch adds the ability to document metrics, and have them validated against metrics that are being measured by the test. The definitions can be done in any "level" of the configuration (test, suite, global), however there can't be any duplicate definitions for metrics (including their aliases).

Warnings are output for the following:

• Duplicate metrics definitions.
• Undocumented metrics that are measured (alerting for Raptor).
• Documented, but unmeasured metrics.

Changes to the tests are to make them work with the changes done in this patch.

Comment 4

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1772015 - [perfdocs] Add metric documentation, and validation to Raptor. r?#perftest

This patch adds the ability to document metrics, and have them validated against metrics that are being measured by the test. The definitions can be done in any "level" of the configuration (test, suite, global), however there can't be any duplicate definitions for metrics (including their aliases).

Warnings are output for the following:

• Duplicate metrics definitions.
• Undocumented metrics that are measured (alerting for Raptor).
• Documented, but unmeasured metrics.

Changes to the tests are to make them work with the changes done in this patch.

Comment 5

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1772015 - [perfdocs] Add metric documentation, and validation to Raptor. r?#perftest

This patch adds the ability to document metrics, and have them validated against metrics that are being measured by the test. The definitions can be done in any "level" of the configuration (test, suite, global), however there can't be any duplicate definitions for metrics (including their aliases).

Warnings are output for the following:

• Duplicate metrics definitions.
• Undocumented metrics that are measured (alerting for Raptor).
• Documented, but unmeasured metrics.

Changes to the tests are to make them work with the changes done in this patch.

Comment 6

[Myeongjun Go]

Attachment: Bug 1772015 - Add starter docs for metrics. r=sparky

This patch adds some starter documentation on metrics and our tests that is unfinished. It also adds metric definitions which take care of the perfdoc failures generated in previous patches in the series.

Comment 7

[Myeongjun Go]

Attachment: Bug 1772015 - [perfdocs] Add metric documentation, and validation to Raptor. r=sparky

This patch adds the ability to document metrics, and have them validated against metrics that are being measured by the test. The definitions can be done in any "level" of the configuration (test, suite, global), however there can't be any duplicate definitions for metrics (including their aliases).

Warnings are output for the following:

• Duplicate metrics definitions.
• Undocumented metrics that are measured (alerting for Raptor).
• Documented, but unmeasured metrics.

Changes to the tests are to make them work with the changes done in this patch.

Comment 8

[Myeongjun Go]

Attachment: Bug 1772015 - [perfdocs] Add tests for new metrics field. r=sparky

This patch adds some new tests for the metrics field. It also fixes up some issues in the existing tests which came from mock.Mock being used outside of a context manager. Other changes are updates to the existing tests.

Depends on D156851

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/26f96cf4114d
[perfdocs] Add metric documentation, and validation to Raptor. r=perftest-reviewers,aglavic
https://hg.mozilla.org/integration/autoland/rev/e36977a9e313
[perfdocs] Add tests for new metrics field. r=sparky,perftest-reviewers
https://hg.mozilla.org/integration/autoland/rev/0788b29168f8
Add starter docs for metrics. r=sparky,perftest-reviewers

Comment 10

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/26f96cf4114d
https://hg.mozilla.org/mozilla-central/rev/e36977a9e313
https://hg.mozilla.org/mozilla-central/rev/0788b29168f8