1772829 - [meta] Improve comment 0 and alert bug for developers

Status: NEW

(Testing :: Performance, task, P2)

Description

[Greg Mierzwinski [:sparky]]

We need to improve out comment 0 or alert template here. This includes adding more info about our tools, metrics, what the metrics means (and how they're obtained, etc.), how to debug the regression, what the extra options mean (e.g. cold, and warm), and a list of Do and don'ts (e.g. not using a VM to get representative numbers).

Comment 1

[Greg Mierzwinski [:sparky]]

We should also add documentation for how to run the profiler (locally, through try from ./mach try, and through the generate profile button).

Comment 2

[Greg Mierzwinski [:sparky]]

Part of this should involve looking into this page and moving it to perfdocs (along with a major upgrade): https://wiki.mozilla.org/TestEngineering/Performance/RegressionBugsHandling

Comment 3

[Greg Mierzwinski [:sparky]]

Here's another page: https://wiki.mozilla.org/TestEngineering/Performance/FAQ

Comment 4

[Kimberly Sereduck :kimberlythegeek]

Consider adding a link to Perfherder Compare comparing before/after results

Comment 5

[Greg Mierzwinski [:sparky]]

See this comment about following the 3-day backout policy: https://bugzilla.mozilla.org/show_bug.cgi?id=1776421#c1

Comment 6

[Julien Wajsberg [:julienw]]

hey folks,
This came from a discussion this morning with florian about this alert: bug 1993725.

While it improved a lot in the past months, we thought of some additional possible improvements. I think most of them are one-line patches :-)

1. Add a link to perfcompare directly from the bugzilla comment 0.
2. Add another title after the tables, before the rest of the text. Otherwise visually it looks like it belongs to the "improvements" part and as a reader I want to skip it despite there are valuable information there.
3. Have perfcompare links use the "sort by delta" parameter (both the one in the alert and the one that would be added from the 1st item in this list). This is as simple as adding &sort=delta|desc to the URL.
4. Would it be possible to have replicates for these runs (I assume they are backfills)?
5. The profiler link could link directly to the right profile inside the zip (cold vs warm) (not sure how easy that one is though).
6. The term "loadtime" isn't great because we don't see it in the profile -- I assumed this is DocumentLoad in the profile? At least that should be mentioned somewhere, possibly in the test description on firefox-source-docs?
7. The term "loadtime" could have a link to https://firefox-source-docs.mozilla.org/testing/perfdocs/raptor-metrics.html#load-time (both in the bugzilla comment 0 and in the alert page)
8. It's not obvious to me that clicking on the test name will lead to the test documentation. I was gonna suggest to add the links below for the displayed tests, but this isn't great either. So not sure what to do. Maybe add a (doc) text right after, and put the link on this only?
9. There are 2 different regressions from otherwise identical tests. My understanding is that they were merged from another alert. It's strange that there are no link on the second regression line's values. Although these links are of questionable value... I'm not sure I'd keep them in comment 0, the could be more useful on the alert page.

Comment 7

[Acasandrei Beatrice (needinfo me)]

I have addressed some of the feedback provided by Julien in PR #9156.

• Add PerfCompare link to regression bug in comment 0 (1.)
• Add another title after the tables, before the rest of the text (2.)
• Have the Perfcompare links use the "sort by delta" parameter (3.)
• Added a specific (doc) text link next to each displayed test (8.)