[perfdocs] Enable dynamic metric documentation with Raptor as the first harness to use it
Categories
(Testing :: Performance, task, P3)
Tracking
(firefox127 fixed)
| Tracking | Status | |
|---|---|---|
| firefox127 | --- | fixed |
People
(Reporter: sparky, Assigned: sparky)
References
(Blocks 2 open bugs)
Details
(Whiteboard: [fxp])
Attachments
(5 files)
This bug is for adding dynamic metric documentation generation to perfdocs, and using it in the Raptor test documentation.
|
||
Updated•5 months ago
|
|
Assignee | |
Comment 1•5 months ago
|
||
This patch removes the metric documentation definition ability from the suite/test levels. This is done because there's no case where a metric would need to be added to a single test, or a suite versus the global level. The tests that contain the metric are found automatically as well. Furthermore, it's already disallowed to have multiple definitions of a metric so having various levels makes no difference, and complicates the verification code. At the same time the valid metrics are gathered so that we can use them when building the metrics documentation. This prevents us from having to parse the YAML, and test lists a second time.
|
||
Updated•5 months ago
|
|
|
Assignee | |
Comment 2•5 months ago
|
||
This patch adds the ability to document metrics dynamically. The metrics are documented in the config YAML file for each framework, and then defined in a metrics.rst file. Two things are expected to be found in the framework docs now if they want to use dynamic metric documentation:
- A metrics.rst file that contains a
{metrics_documentation}entry somewhere to insert the built documentation. - A
{metrics_rst_name}entry somewhere in the top-level/index.rst file so that a link can be produced to the metrics documentation.
The metrics documentation is specific to each framework. See the schema for the metrics for what is expected in this field. At the same time, the dynamic documentation for Raptor is generated here. Each metric in Raptor contains a list of all the tests that use (alert on) the specific metric. The tests link back to the test definition, and the metrics defined in the tests also link to the metric definitions.
Depends on D206949
|
|
Assignee | |
Comment 3•5 months ago
|
||
This patch fixes some unit tests, and adds a new one for the metric documentation generation.
Depends on D206950
|
|
Assignee | |
Comment 4•5 months ago
|
||
This patch adds the final missing metrics that alert to the Raptor documentation.
Depends on D206951
|
|
Assignee | |
Comment 5•5 months ago
|
||
This patch updates the perfdocs linter documentation to contain the new config schema, and information about how to add documentation for the metrics.
Depends on D206952
Pushed by gmierz2@outlook.com: https://hg.mozilla.org/integration/autoland/rev/532f5ae4f6cf Remove metrics definition from suite/test levels, and gather valid metrics. r=perftest-reviewers,aglavic DONTBUILD https://hg.mozilla.org/integration/autoland/rev/3c0511d7c422 Add dynamic metric documentation to perfdocs. r=perftest-reviewers,aglavic DONTBUILD https://hg.mozilla.org/integration/autoland/rev/e186be4fad31 Add unit tests for perfdocs metric documentation generation. r=perftest-reviewers,afinder DONTBUILD https://hg.mozilla.org/integration/autoland/rev/2bcd0e24c735 Add missing metrics to Raptor documentation. r=perftest-reviewers,kshampur DONTBUILD https://hg.mozilla.org/integration/autoland/rev/32f820922a96 Update perfdocs linter documentation. r=perftest-reviewers,kshampur DONTBUILD
https://hg.mozilla.org/mozilla-central/rev/532f5ae4f6cf
https://hg.mozilla.org/mozilla-central/rev/3c0511d7c422
https://hg.mozilla.org/mozilla-central/rev/e186be4fad31
https://hg.mozilla.org/mozilla-central/rev/2bcd0e24c735
https://hg.mozilla.org/mozilla-central/rev/32f820922a96
Description
•