Closed Bug 752002 Opened 8 years ago Closed 8 years ago

Don't send talos perf regression email to individual patch authors for giant merges

Categories

(Release Engineering :: General, defect, P3)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mbrubeck, Assigned: catlee)

References

Details

(Whiteboard: [regressions])

Attachments

(1 file)

We are now sending regression emails to changeset authors/committers, in addition to the dev-tree-management mailing list.  However, this generates a large amount of noise because a lot of these emails are for merges from one branch to another and so are generally not actionable for individual patch authors.

To increase the signal:noise ratio (and hopefully stop some developers from filtering all regression messages to the trash), perhaps we should *not* email individuals if the number of separate changesets or authors is higher than some threshold.  (I'm thinking around 50 changesets, or 20 unique patch authors.)
Component: Release Engineering → Release Engineering: Automation (General)
QA Contact: release → catlee
Assignee: nobody → catlee
Priority: -- → P3
Whiteboard: [regressions]
Blocks: 761484
If we fix this bug, we should change the e-mails so as to break filters which may have been applied to block these e-mails.
I don't know how to figure out if a changeset on hg.m.o is a merge or not, so am going to use an upper bound of people to notify.

Let's pick 20. If there are more than 20 people on the list, then what?
- Just send to the newsgroup?
- Send to the newsgroup + the lucky 20 people?
- Send to the newsgroup with some indication that a large number of people are involved?
> - Send to the newsgroup + the lucky 20 people?

Not this, please!  Suppose we have a merge involving 30 people.  Now 2/3 of those will get a useless message.

> I don't know how to figure out if a changeset on hg.m.o is a merge or not

How are you looking?
(In reply to Justin Lebar [:jlebar] from comment #3)
> > I don't know how to figure out if a changeset on hg.m.o is a merge or not
> 
> How are you looking?

via the json-pushes api on hg.m.o
http://hg.mozilla.org/mozilla-central/json-pushes?full=1&changeset=6873940886b2

it doesn't include any parent information for the nodes.
I don't think we really need to know if they are merges or not.  Any changeset with more than 10 or whatever authors is going to have its Talos numbers ignored by anybody who gets it.  Too much diffusion of responsibility.

In theory, I could imagine there being people who'd want to get emails for Talos regressions on large changesets ("Uh oh, a regression made it to m-c from inbound!"), but I think at this point anybody who cares that much is going to pay attention to the tree management group and not really need any additional mailing.
(In reply to Chris AtLee [:catlee] from comment #2)
> Let's pick 20. If there are more than 20 people on the list, then what?
> - Just send to the newsgroup?
> - Send to the newsgroup with some indication that a large number of people are involved?

I think either of these would be good.  I don't think I personally have a use for the "large number of people" indication, but if someone else does then I have no objection.
I think we should:
* Send all mails to the newsgroup (no need for large group annotation)
* If number of authors < 5, then email the individual authors

Anything higher than 5-10 authors and people will just ignore the emails. We can always raise the figure later once people start to trust talos a little more.

Also, don't forget of the time there will either be 1-3 authors, or else it is a merge (and the regression would have triggered emails on the originating tree, so no need for more of them). Checkin-neededs being perhaps the one exception to this - however even then there normally won't be any more than half a dozen authors in a push (though I know RyanVM likes to push as many as possible at once; but I personally think this is something we should avoid).
(In reply to Ed Morley [:edmorley] from comment #7)
> I think we should:
> * Send all mails to the newsgroup (no need for large group annotation)
> * If number of authors < 5, then email the individual authors

Should have read:
"then email the individual authors *as well*"
Comment on attachment 658185 [details] [diff] [review]
change email_authors to max_email_authors, and default to 5

Looks fine to me.
Attachment #658185 - Flags: review?(nthomas) → review+
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/754afcc171500a0b7d9fe55666b32f9c2c114b64
Bug 752002: change email_authors to max_email_authors and default to 5. r=nthomas
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.