alert manager should generate a template for filing bugs

RESOLVED FIXED

Status

Testing
General
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: jmaher, Assigned: mishravikas, Mentored)

Tracking

Trunk
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

3 years ago
Whenever a human is involved it introduces error, whenever we rely on automation there is less trust in the system.  

Right now when we file a bug that notifies a developer about a performance regression, we manually do a lot of things (some we skip, but we shouldn't):
* create the subject
* add a link to a graph
* add a link to tbpl
* add a link to the pushlog
* mention how to test this on try/locally
* link to the test on the talos wiki
* depend on all bugs in the commit, as well as the specific tracking bug
* cc the patch authors and reviewers
* outline the list of improvements/regressions (can be a link to alertmanager) with notable mentions in the bug comment
* mention the expectations about responding to the bug

All of these things can be automated- but lets take it slow at first.  I would like alert manager to take the data it has and start a bug to be filed so we have the title and comment containing the information.  

Ideally we would have a template and then it can be automatically filled in with data from the database.  I presume we can get a bug partially filled in automatically.
(Reporter)

Comment 1

3 years ago
let me expand here a bit.

First off, I think a link for each revision would be useful.  right now we don't have any information at the revision specifically, it is all related to the alert.  In the case of filing bugs, we always file them one bug per revision.

While in a perfect world we would just click a link and a bug would be filed, I think that is too problematic- maybe baby steps at first.

What I see here is a link that would generate a popup with information that we can cut/paste.  If we can find a way to send it to enter_bug.cgi and let us edit it before filing it, I would approve :)  For example a shortcut to file a talos bug is:
https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Talos


information we care about:
-------------------------------------------------------------------------------------------------------
product: testing
component: talos
Summary: <%> <platform> <test> regression on <branch> (v.<xx>) on <date> from push <revision>
Description:
We have verified a regression, here is a list to all the regressions and improvements that we know of so far:
http://alertmanager.allizom.org:8080/alerts.html?rev=ed9225015e22&showAll=1&testIndex=0&platIndex=0

Viewing the above alerts in alert manager you can view each alert as well as a link to the graph showing the regression.  There is also a link to treeherder showing the jobs in a pushlog format.

To learn more about this test, please visit:
https://wiki.mozilla.org/Buildbot/Talos/Tests#CanvasMark <- note we would need to substitute the correct name here

to run this locally, follow these instructions to get setup:
https://wiki.mozilla.org/Buildbot/Talos/Running#Running_locally_-_Source_Code

and then run:
talos --develop -e <path>/firefox -a canvasmark

If you want to push to try, you can use this syntax:
try: -b o -p linux -u none -t chromez

As the patch author we need your feedback to help us determine what the next steps are:
* If this doesn't seem possible that it could be related to your bug, let us know ASAP
* If this is possible that you caused a regression, but not all of this, let us know ASAP
* Otherwise, we would like to know by <Tuesday> if:
** this is expected, and we should accept the regression. Please provide documentation and we will close this bug
** this is unexpected, but is limited in scope and scale- to investigate and fix this would be not worthwhile based on how long it would take to sort out
** this is unexpected, but after a brief look at the code it is likely you could fix this; do let us know and provide a general timeline
** this is unexpected and we are not sure what to do, we should backout for now.


-------------------------------------------------------------------------------------------------------


the wording might need to be slightly adjusted, and filling in some of the blanks might take a few back and forth comments.

an example subject for:
http://alertmanager.allizom.org:8080/alerts.html?rev=ed9225015e22&showAll=1&testIndex=0&platIndex=0

would be:
4-21% Linux*/Win* CART/TART Regression on Inbound (v.38) on Jan 22 from push ed9225015e22
(Assignee)

Comment 2

3 years ago
I'm on it
(Reporter)

Updated

3 years ago
Assignee: nobody → vikasmishra95
(Reporter)

Comment 3

3 years ago
this is fixed in:
https://github.com/jmaher/alert_manager/pull/130

I imagine as we use this there will be little tweaks, but this gets us most of the way there!
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.