752002 - Don't send talos perf regression email to individual patch authors for giant merges

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Matt Brubeck (:mbrubeck)]

We are now sending regression emails to changeset authors/committers, in addition to the dev-tree-management mailing list.  However, this generates a large amount of noise because a lot of these emails are for merges from one branch to another and so are generally not actionable for individual patch authors.

To increase the signal:noise ratio (and hopefully stop some developers from filtering all regression messages to the trash), perhaps we should *not* email individuals if the number of separate changesets or authors is higher than some threshold.  (I'm thinking around 50 changesets, or 20 unique patch authors.)

Comment 1

[Justin Lebar (not reading bugmail)]

If we fix this bug, we should change the e-mails so as to break filters which may have been applied to block these e-mails.

Comment 2

[Chris AtLee [:catlee]]

I don't know how to figure out if a changeset on hg.m.o is a merge or not, so am going to use an upper bound of people to notify.

Let's pick 20. If there are more than 20 people on the list, then what?
- Just send to the newsgroup?
- Send to the newsgroup + the lucky 20 people?
- Send to the newsgroup with some indication that a large number of people are involved?

Comment 3

[Justin Lebar (not reading bugmail)]

> - Send to the newsgroup + the lucky 20 people?

Not this, please!  Suppose we have a merge involving 30 people.  Now 2/3 of those will get a useless message.

> I don't know how to figure out if a changeset on hg.m.o is a merge or not

How are you looking?

Comment 4

[Chris AtLee [:catlee]]

(In reply to Justin Lebar [:jlebar] from comment #3)
> > I don't know how to figure out if a changeset on hg.m.o is a merge or not
> 
> How are you looking?

via the json-pushes api on hg.m.o
http://hg.mozilla.org/mozilla-central/json-pushes?full=1&changeset=6873940886b2

it doesn't include any parent information for the nodes.

Comment 5

[Andrew McCreight [:mccr8]]

I don't think we really need to know if they are merges or not.  Any changeset with more than 10 or whatever authors is going to have its Talos numbers ignored by anybody who gets it.  Too much diffusion of responsibility.

In theory, I could imagine there being people who'd want to get emails for Talos regressions on large changesets ("Uh oh, a regression made it to m-c from inbound!"), but I think at this point anybody who cares that much is going to pay attention to the tree management group and not really need any additional mailing.

Comment 6

[Matt Brubeck (:mbrubeck)]

(In reply to Chris AtLee [:catlee] from comment #2)
> Let's pick 20. If there are more than 20 people on the list, then what?
> - Just send to the newsgroup?
> - Send to the newsgroup with some indication that a large number of people are involved?

I think either of these would be good.  I don't think I personally have a use for the "large number of people" indication, but if someone else does then I have no objection.

Comment 7

[Ed Morley [:emorley]]

I think we should:
* Send all mails to the newsgroup (no need for large group annotation)
* If number of authors < 5, then email the individual authors

Anything higher than 5-10 authors and people will just ignore the emails. We can always raise the figure later once people start to trust talos a little more.

Also, don't forget of the time there will either be 1-3 authors, or else it is a merge (and the regression would have triggered emails on the originating tree, so no need for more of them). Checkin-neededs being perhaps the one exception to this - however even then there normally won't be any more than half a dozen authors in a push (though I know RyanVM likes to push as many as possible at once; but I personally think this is something we should avoid).

Comment 8

[Ed Morley [:emorley]]

(In reply to Ed Morley [:edmorley] from comment #7)
> I think we should:
> * Send all mails to the newsgroup (no need for large group annotation)
> * If number of authors < 5, then email the individual authors

Should have read:
"then email the individual authors *as well*"

Comment 9

[Chris AtLee [:catlee]]

Attachment: change email_authors to max_email_authors, and default to 5

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 658185 [details] [diff] [review]
change email_authors to max_email_authors, and default to 5

Looks fine to me.

Comment 11

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/754afcc171500a0b7d9fe55666b32f9c2c114b64
Bug 752002: change email_authors to max_email_authors and default to 5. r=nthomas