535023 - have talos regression tracker auto-post to bugzilla

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P3)

Description

[alice nodelman [:alice] [:anode]]

The regression script knows changesets, so it should be able to determine a list of bugs and then post in the whiteboard of each bug in a given range.  The whiteboard post would be something along the lines of "regression-suspect" and a comment would also be added to the effect "this bug is part of the regression range for [Testname] on [branch] found in [graph], once cleared of suspicion the white board comment may be removed".

We should probably run this by sheriffs as well to determine what format they would like best.

Comment 1

[Chris Cooper [:coop] (he/him)]

Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Comment 2

[Chris AtLee [:catlee]]

Beltzner just asked if we could do this, it would save him a lot of time since he's doing it manually right now.

Comment 3

[Mike Beltzner [:beltzner, not reading bugmail]]

To be clear, I'm not commenting on bugs, I'm cc'ing the committer, bug owners and patch authors on the dev-tree-management emails to make sure that they see them. This allows me to get a bunch of people tagged in a range to look.

I think Alice's proposal in comment 0 is probably better, but cc-on-email is a nice second-best.

Comment 4

[Chris AtLee [:catlee]]

Suspected!

Comment 5

[Chris AtLee [:catlee]]

Oops...apparently using api-dev.bugzilla.mozilla.org affects the real-live bugzilla!

Comment 6

[Chris AtLee [:catlee]]

Any comments on the message format here:

https://bugzilla.mozilla.org/show_bug.cgi?id=test#c28

Comment 7

[Armen [:armenzg]]

(In reply to comment #6)
> Any comments on the message format here:
> 
> https://bugzilla.mozilla.org/show_bug.cgi?id=test#c28

I love that you are using mzl.la! The message is clear and precise.

Two questions:
* would it be helpful to add a link to the changeset?
* would adding the branch name help?

Just three nits:
* I would use "increase" instead of "went"
* Besides the numeric values can we add "s" or "ms"? I don't know what measurement unit is
* add "+" as a preffix to the percentage number

> This bug is part of the regression range for DHTML on Firefox found in
> http://mzl.la/cOfpll.
> 
> The test value went from 653.294 to 655.735 (0.374%)
> 
> Once cleared of suspicion the whiteboard comment may be removed.

Comment 8

[Chris AtLee [:catlee]]

(In reply to comment #7)
> (In reply to comment #6)
> > Any comments on the message format here:
> > 
> > https://bugzilla.mozilla.org/show_bug.cgi?id=test#c28
> 
> I love that you are using mzl.la! The message is clear and precise.
> 
> Two questions:
> * would it be helpful to add a link to the changeset?

Hmmm, I'm not sure.

> * would adding the branch name help?

It's already in there, https://bugzilla.mozilla.org/show_bug.cgi?id=11383#c36, "SVG on Firefox"

> Just three nits:
> * I would use "increase" instead of "went"

no problem

> * Besides the numeric values can we add "s" or "ms"? I don't know what
> measurement unit is

I have no idea what the units are either...they're not defined anywhere

> * add "+" as a preffix to the percentage number

done!

Comment 9

[Chris AtLee [:catlee]]

How's this: https://bugzilla.mozilla.org/show_bug.cgi?id=11383#c53

Comment 10

[Armen [:armenzg]]

Looks good :)

Comment 11

[Chris AtLee [:catlee]]

Attachment: lots...including posting comments to bugzilla

So, my current commit message, which explains a lot of what's going on, is:

Refactoring:

* Removed analysis.cfg and added analysis.cfg.template to be used as an example
  config file
* Deleted unused analysis methods
* Removed unused threading support
* Removed unused graphapi support
* Removed html email notifications

Bug fixes:

* Made variance calculate more accurate
* Make sure that we still print urls even if shortening fails
* Compare previous mean vs new mean to determine if a change is a regression or
 not

Enhancements:

* Used new-style classes and slots for PerfDatum class, drastically reducing
  memory usage
* Bug 535023: Add ability to post comments in bugs related to regressions
* Get more data from pushlog, not just the push time
* Bug 492983: Add links to each revision in the suspected checkin range, along
  with authors and change comments, and generally improved email formatting
* Add ability to notify push and change authors of regressions
* Use statusdb for finding inactive machines (faster, more accurate)
* Cache the latest test_runs.id seen, and use that as the starting point for
  queries next time.  Drastically improves time to run.

Comment 12

[Reed Loden [:reed]]

Comment on attachment 475105 [details] [diff] [review]
lots...including posting comments to bugzilla

>+# How to talk to bugzilla
>+#bz_api = https://api-dev.bugzilla.mozilla.org/0.6.1

Are you sure you want to hardcode a specific BzAPI version? Gerv regularly deprecates and removes old versions, so you might find yourself not working at all in the not-to-distant future. While there is the possibility of something breaking, Gerv works to keep backwards compatibility when/where he can. So, might be better to use 'latest/'. Just a thought...

>+# For testing, which bug to add comments to instead of the real thing
>+#bz_bug_override = 11383

Please do NOT use production Bugzilla (bmo) for testing. See https://wiki.mozilla.org/Bugzilla:REST_API#Servers for the test server URIs.

Comment 13

[Chris AtLee [:catlee]]

(In reply to comment #12)
> Comment on attachment 475105 [details] [diff] [review]
> lots...including posting comments to bugzilla
> 
> >+# How to talk to bugzilla
> >+#bz_api = https://api-dev.bugzilla.mozilla.org/0.6.1
> 
> Are you sure you want to hardcode a specific BzAPI version? Gerv regularly
> deprecates and removes old versions, so you might find yourself not working at
> all in the not-to-distant future. While there is the possibility of something
> breaking, Gerv works to keep backwards compatibility when/where he can. So,
> might be better to use 'latest/'. Just a thought...

It seems safer to work against a known api version than against something that can change without me knowing.

> >+# For testing, which bug to add comments to instead of the real thing
> >+#bz_bug_override = 11383
> 
> Please do NOT use production Bugzilla (bmo) for testing. See
> https://wiki.mozilla.org/Bugzilla:REST_API#Servers for the test server URIs.

That's great, except the testing servers are inaccessible to mere mortals.

Comment 14

[Reed Loden [:reed]]

(In reply to comment #13)
> > Are you sure you want to hardcode a specific BzAPI version? Gerv regularly
> > deprecates and removes old versions, so you might find yourself not working at
> > all in the not-to-distant future. While there is the possibility of something
> > breaking, Gerv works to keep backwards compatibility when/where he can. So,
> > might be better to use 'latest/'. Just a thought...
> 
> It seems safer to work against a known api version than against something that
> can change without me knowing.

Sure, I understand. Just keep in mind that Gerv regularly deprecates old versions without much warning.

> > Please do NOT use production Bugzilla (bmo) for testing. See
> > https://wiki.mozilla.org/Bugzilla:REST_API#Servers for the test server URIs.
> 
> That's great, except the testing servers are inaccessible to mere mortals.

Recently fixed! https://landfill.bugzilla.org/bzapi_sandbox/ is the new testing server that is open to all. See https://wiki.mozilla.org/Bugzilla:REST_API#Servers for the info.

Comment 15

[Chris AtLee [:catlee]]

(In reply to comment #14)
> (In reply to comment #13)
> > > Are you sure you want to hardcode a specific BzAPI version? Gerv regularly
> > > deprecates and removes old versions, so you might find yourself not working at
> > > all in the not-to-distant future. While there is the possibility of something
> > > breaking, Gerv works to keep backwards compatibility when/where he can. So,
> > > might be better to use 'latest/'. Just a thought...
> > 
> > It seems safer to work against a known api version than against something that
> > can change without me knowing.
> 
> Sure, I understand. Just keep in mind that Gerv regularly deprecates old
> versions without much warning.
> 
> > > Please do NOT use production Bugzilla (bmo) for testing. See
> > > https://wiki.mozilla.org/Bugzilla:REST_API#Servers for the test server URIs.
> > 
> > That's great, except the testing servers are inaccessible to mere mortals.
> 
> Recently fixed! https://landfill.bugzilla.org/bzapi_sandbox/ is the new testing
> server that is open to all. See
> https://wiki.mozilla.org/Bugzilla:REST_API#Servers for the info.

Woo!

Comment 16

[Mike Taylor [:bear]]

Comment on attachment 475105 [details] [diff] [review]
lots...including posting comments to bugzilla

apologies for the lag in clicking "save"

Comment 17

[Chris AtLee [:catlee]]

Comment on attachment 475105 [details] [diff] [review]
lots...including posting comments to bugzilla

318:da91ebe9768f

Comment 18

[Chris AtLee [:catlee]]

So, I think this needs to avoid posting comments / whiteboard changes to the bug if someone has removed the whiteboard tag.

See e.g. 531056.  The tags were added, removed, and added again because the same regression was detected on another platform.

Comment 19

[Chris AtLee [:catlee]]

Also, should these bug comments be enabled only for certain branches?

Comment 20

[Chris AtLee [:catlee]]

Attachment: Improve bug commenting logic

This adds a per-branch option that must be set in the config to indicate if regressions on that branch will generate bug comments.

If a bug is implicated in a regression range, we now look through all the bug's comments to see if we previously implicated it for the same os/branch/revision range.  If we did, we won't post another comment.

Comment 21

[Robert Sayre]

this is a terrible idea. it just spammed the living daylights out of the js team. posting to the newsgroup is quite enough, thanks.

Comment 22

[Chris AtLee [:catlee]]

As per dev.tree-management discussion.

Comment 23

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/fd01054c03a543c50b3f7a132c5505325e5e420b
Bugs 535023, 492983: various improvements and fixes. r=bear

Refactoring:

* Removed analysis.cfg and added analysis.cfg.template to be used as an example
  config file
* Deleted unused analysis methods
* Removed unused threading support
* Removed unused graphapi support
* Removed html email notifications

Bug fixes:

* Made variance calculate more accurate
* Make sure that we still print urls even if shortening fails
* Compare previous mean vs new mean to determine if a change is a regression or
 not

Enhancements:

* Used new-style classes and slots for PerfDatum class, drastically reducing
  memory usage
* Bug 535023: Add ability to post comments in bugs related to regressions
* Get more data from pushlog, not just the push time
* Bug 492983: Add links to each revision in the suspected checkin range, along
  with authors and change comments, and generally improved email formatting
* Add ability to notify push and change authors of regressions
* Use statusdb for finding inactive machines (faster, more accurate)
* Cache the latest test_runs.id seen, and use that as the starting point for
  queries next time.  Drastically improves time to run.

--HG--
branch : 1.0