have talos regression tracker auto-post to bugzilla

RESOLVED WONTFIX

Status

Release Engineering
General
P3
normal
RESOLVED WONTFIX
8 years ago
2 years ago

People

(Reporter: alice, Assigned: catlee)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [talos] [automation][oldbug])

Attachments

(2 attachments)

(Reporter)

Description

8 years ago
The regression script knows changesets, so it should be able to determine a list of bugs and then post in the whiteboard of each bug in a given range.  The whiteboard post would be something along the lines of "regression-suspect" and a comment would also be added to the effect "this bug is part of the regression range for [Testname] on [branch] found in [graph], once cleared of suspicion the white board comment may be removed".

We should probably run this by sheriffs as well to determine what format they would like best.

Updated

8 years ago
Component: Release Engineering → Release Engineering: Future
Priority: -- → P3
Whiteboard: [talos] [automation]

Comment 1

8 years ago
Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.
Component: Release Engineering: Future → Release Engineering
(Assignee)

Comment 2

7 years ago
Beltzner just asked if we could do this, it would save him a lot of time since he's doing it manually right now.
To be clear, I'm not commenting on bugs, I'm cc'ing the committer, bug owners and patch authors on the dev-tree-management emails to make sure that they see them. This allows me to get a bunch of people tagged in a range to look.

I think Alice's proposal in comment 0 is probably better, but cc-on-email is a nice second-best.

Updated

7 years ago
Assignee: nobody → catlee
Whiteboard: [talos] [automation] → [talos] [automation][oldbugtriage]
(Assignee)

Updated

7 years ago
Whiteboard: [talos] [automation][oldbugtriage] → [talos] [automation][oldbug]
(Assignee)

Updated

7 years ago
Whiteboard: [talos] [automation][oldbug] → [talos] [automation][oldbug][regression-suspect]
(Assignee)

Updated

7 years ago
Whiteboard: [talos] [automation][oldbug][regression-suspect] → [talos] [automation][oldbug]
(Assignee)

Updated

7 years ago
Whiteboard: [talos] [automation][oldbug] → [talos] [automation][oldbug][regression-suspect]
(Assignee)

Comment 4

7 years ago
Suspected!
(Assignee)

Updated

7 years ago
Whiteboard: [talos] [automation][oldbug][regression-suspect] → [talos] [automation][oldbug]
(Assignee)

Comment 5

7 years ago
Oops...apparently using api-dev.bugzilla.mozilla.org affects the real-live bugzilla!
(Assignee)

Comment 6

7 years ago
Any comments on the message format here:

https://bugzilla.mozilla.org/show_bug.cgi?id=test#c28

Comment 7

7 years ago
(In reply to comment #6)
> Any comments on the message format here:
> 
> https://bugzilla.mozilla.org/show_bug.cgi?id=test#c28

I love that you are using mzl.la! The message is clear and precise.

Two questions:
* would it be helpful to add a link to the changeset?
* would adding the branch name help?

Just three nits:
* I would use "increase" instead of "went"
* Besides the numeric values can we add "s" or "ms"? I don't know what measurement unit is
* add "+" as a preffix to the percentage number

> This bug is part of the regression range for DHTML on Firefox found in
> http://mzl.la/cOfpll.
> 
> The test value went from 653.294 to 655.735 (0.374%)
> 
> Once cleared of suspicion the whiteboard comment may be removed.
(Assignee)

Comment 8

7 years ago
(In reply to comment #7)
> (In reply to comment #6)
> > Any comments on the message format here:
> > 
> > https://bugzilla.mozilla.org/show_bug.cgi?id=test#c28
> 
> I love that you are using mzl.la! The message is clear and precise.
> 
> Two questions:
> * would it be helpful to add a link to the changeset?

Hmmm, I'm not sure.

> * would adding the branch name help?

It's already in there, https://bugzilla.mozilla.org/show_bug.cgi?id=11383#c36, "SVG on Firefox"

> Just three nits:
> * I would use "increase" instead of "went"

no problem

> * Besides the numeric values can we add "s" or "ms"? I don't know what
> measurement unit is

I have no idea what the units are either...they're not defined anywhere

> * add "+" as a preffix to the percentage number

done!
(Assignee)

Comment 9

7 years ago
How's this: https://bugzilla.mozilla.org/show_bug.cgi?id=11383#c53

Comment 10

7 years ago
Looks good :)
(Assignee)

Comment 11

7 years ago
Created attachment 475105 [details] [diff] [review]
lots...including posting comments to bugzilla

So, my current commit message, which explains a lot of what's going on, is:

Refactoring:

* Removed analysis.cfg and added analysis.cfg.template to be used as an example
  config file
* Deleted unused analysis methods
* Removed unused threading support
* Removed unused graphapi support
* Removed html email notifications

Bug fixes:

* Made variance calculate more accurate
* Make sure that we still print urls even if shortening fails
* Compare previous mean vs new mean to determine if a change is a regression or
 not

Enhancements:

* Used new-style classes and slots for PerfDatum class, drastically reducing
  memory usage
* Bug 535023: Add ability to post comments in bugs related to regressions
* Get more data from pushlog, not just the push time
* Bug 492983: Add links to each revision in the suspected checkin range, along
  with authors and change comments, and generally improved email formatting
* Add ability to notify push and change authors of regressions
* Use statusdb for finding inactive machines (faster, more accurate)
* Cache the latest test_runs.id seen, and use that as the starting point for
  queries next time.  Drastically improves time to run.
Attachment #475105 - Flags: review?(bear)
Comment on attachment 475105 [details] [diff] [review]
lots...including posting comments to bugzilla

>+# How to talk to bugzilla
>+#bz_api = https://api-dev.bugzilla.mozilla.org/0.6.1

Are you sure you want to hardcode a specific BzAPI version? Gerv regularly deprecates and removes old versions, so you might find yourself not working at all in the not-to-distant future. While there is the possibility of something breaking, Gerv works to keep backwards compatibility when/where he can. So, might be better to use 'latest/'. Just a thought...

>+# For testing, which bug to add comments to instead of the real thing
>+#bz_bug_override = 11383

Please do NOT use production Bugzilla (bmo) for testing. See https://wiki.mozilla.org/Bugzilla:REST_API#Servers for the test server URIs.
(Assignee)

Comment 13

7 years ago
(In reply to comment #12)
> Comment on attachment 475105 [details] [diff] [review]
> lots...including posting comments to bugzilla
> 
> >+# How to talk to bugzilla
> >+#bz_api = https://api-dev.bugzilla.mozilla.org/0.6.1
> 
> Are you sure you want to hardcode a specific BzAPI version? Gerv regularly
> deprecates and removes old versions, so you might find yourself not working at
> all in the not-to-distant future. While there is the possibility of something
> breaking, Gerv works to keep backwards compatibility when/where he can. So,
> might be better to use 'latest/'. Just a thought...

It seems safer to work against a known api version than against something that can change without me knowing.

> >+# For testing, which bug to add comments to instead of the real thing
> >+#bz_bug_override = 11383
> 
> Please do NOT use production Bugzilla (bmo) for testing. See
> https://wiki.mozilla.org/Bugzilla:REST_API#Servers for the test server URIs.

That's great, except the testing servers are inaccessible to mere mortals.
(In reply to comment #13)
> > Are you sure you want to hardcode a specific BzAPI version? Gerv regularly
> > deprecates and removes old versions, so you might find yourself not working at
> > all in the not-to-distant future. While there is the possibility of something
> > breaking, Gerv works to keep backwards compatibility when/where he can. So,
> > might be better to use 'latest/'. Just a thought...
> 
> It seems safer to work against a known api version than against something that
> can change without me knowing.

Sure, I understand. Just keep in mind that Gerv regularly deprecates old versions without much warning.

> > Please do NOT use production Bugzilla (bmo) for testing. See
> > https://wiki.mozilla.org/Bugzilla:REST_API#Servers for the test server URIs.
> 
> That's great, except the testing servers are inaccessible to mere mortals.

Recently fixed! https://landfill.bugzilla.org/bzapi_sandbox/ is the new testing server that is open to all. See https://wiki.mozilla.org/Bugzilla:REST_API#Servers for the info.
(Assignee)

Comment 15

7 years ago
(In reply to comment #14)
> (In reply to comment #13)
> > > Are you sure you want to hardcode a specific BzAPI version? Gerv regularly
> > > deprecates and removes old versions, so you might find yourself not working at
> > > all in the not-to-distant future. While there is the possibility of something
> > > breaking, Gerv works to keep backwards compatibility when/where he can. So,
> > > might be better to use 'latest/'. Just a thought...
> > 
> > It seems safer to work against a known api version than against something that
> > can change without me knowing.
> 
> Sure, I understand. Just keep in mind that Gerv regularly deprecates old
> versions without much warning.
> 
> > > Please do NOT use production Bugzilla (bmo) for testing. See
> > > https://wiki.mozilla.org/Bugzilla:REST_API#Servers for the test server URIs.
> > 
> > That's great, except the testing servers are inaccessible to mere mortals.
> 
> Recently fixed! https://landfill.bugzilla.org/bzapi_sandbox/ is the new testing
> server that is open to all. See
> https://wiki.mozilla.org/Bugzilla:REST_API#Servers for the info.

Woo!

Comment 16

7 years ago
Comment on attachment 475105 [details] [diff] [review]
lots...including posting comments to bugzilla

apologies for the lag in clicking "save"
Attachment #475105 - Flags: review?(bear) → review+
(Assignee)

Comment 17

7 years ago
Comment on attachment 475105 [details] [diff] [review]
lots...including posting comments to bugzilla

318:da91ebe9768f
Attachment #475105 - Flags: checked-in+
(Assignee)

Comment 18

7 years ago
So, I think this needs to avoid posting comments / whiteboard changes to the bug if someone has removed the whiteboard tag.

See e.g. 531056.  The tags were added, removed, and added again because the same regression was detected on another platform.
(Assignee)

Comment 19

7 years ago
Also, should these bug comments be enabled only for certain branches?
(Assignee)

Comment 20

7 years ago
Created attachment 479553 [details] [diff] [review]
Improve bug commenting logic

This adds a per-branch option that must be set in the config to indicate if regressions on that branch will generate bug comments.

If a bug is implicated in a regression range, we now look through all the bug's comments to see if we previously implicated it for the same os/branch/revision range.  If we did, we won't post another comment.
Attachment #479553 - Flags: review?(bear)

Updated

7 years ago
Attachment #479553 - Flags: review?(bear) → review+
(Assignee)

Updated

7 years ago
Attachment #479553 - Flags: checked-in+

Comment 21

7 years ago
this is a terrible idea. it just spammed the living daylights out of the js team. posting to the newsgroup is quite enough, thanks.
(Assignee)

Comment 22

7 years ago
As per dev.tree-management discussion.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → WONTFIX
Product: mozilla.org → Release Engineering

Comment 23

2 years ago
Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/fd01054c03a543c50b3f7a132c5505325e5e420b
Bugs 535023, 492983: various improvements and fixes. r=bear

Refactoring:

* Removed analysis.cfg and added analysis.cfg.template to be used as an example
  config file
* Deleted unused analysis methods
* Removed unused threading support
* Removed unused graphapi support
* Removed html email notifications

Bug fixes:

* Made variance calculate more accurate
* Make sure that we still print urls even if shortening fails
* Compare previous mean vs new mean to determine if a change is a regression or
 not

Enhancements:

* Used new-style classes and slots for PerfDatum class, drastically reducing
  memory usage
* Bug 535023: Add ability to post comments in bugs related to regressions
* Get more data from pushlog, not just the push time
* Bug 492983: Add links to each revision in the suspected checkin range, along
  with authors and change comments, and generally improved email formatting
* Add ability to notify push and change authors of regressions
* Use statusdb for finding inactive machines (faster, more accurate)
* Cache the latest test_runs.id seen, and use that as the starting point for
  queries next time.  Drastically improves time to run.

--HG--
branch : 1.0
You need to log in before you can comment on or make changes to this bug.