675004 - TryServer sends 2 "Thanks for your try submission" emails per push

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P5)

Description

[Daniel Holbert [:dholbert]]

Over the past few days, all of the TryServer builds that I've triggered have sent me two "Thanks for your try submission" emails.  They two emails are exactly the same.  e.g. the most recent one:
{
Thanks for your try submission (http://hg.mozilla.org/try/pushloghtml?changeset=aed73454d87a).  It's the best!

Watch http://tbpl.mozilla.org/?tree=Try&rev=aed73454d87a for your results to come in.

Builds and logs will be available at http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-aed73454d87a.

This directory won't be created until the first builds are uploaded, so please be patient.
}

I first saw this on 7/26 (and it's happened on every push since then).

My last "good" TryServer push was on 7/19.  So this duplicate-email issue appears to have been introduced between 7/19 and 7/26.

Comment 1

[Daniel Holbert [:dholbert]]

(This appears to only affect the initial "Thanks" email.  I've only been getting getting 1 copy of the "build completed" emails (good).)

Comment 2

[Daniel Holbert [:dholbert]]

jdm says he's hit this, too, so it's not just me.

Comment 3

[Lukas Blakk [:lsblakk] use ?needinfo]

those come from http://hg.mozilla.org/build/buildbotcustom/file/tip/status/generators.py whereas the rest of the emails are from try_mailer.py so something is running generators.py twice perhaps?

Comment 4

[Aki Sasaki (not active)]

Has this been fixed?
I pushed to try and only got one email.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

WFM too.

Comment 6

[Daniel Holbert [:dholbert]]

Yup -- I haven't seen this in quite a while, & I forgot that the bug existed. :)

Comment 7

[Nathan Froyd [:froydnj]]

Saw this with five pushes today and nthomas asked me to file it.  A push at 12:54PM Eastern only sent one email; a push at 2:25PM Eastern sent two.

Example headers:

Return-Path: [e-mail]
Received: from zmmta1.mail.corp.phx1.mozilla.com (LHLO
 zmmta1.mail.corp.phx1.mozilla.com) (10.20.77.21) by
 zmmbox6.mail.corp.phx1.mozilla.com with LMTP; Wed, 11 Sep 2013 11:25:18
 -0700 (PDT)
Received: from zmmta1.mail.corp.phx1.mozilla.com (localhost6.localdomain [127.0.0.1])
	by zmmta1.mail.corp.phx1.mozilla.com (Postfix) with ESMTP id D16461063E3
	for <[e-mail]>; Wed, 11 Sep 2013 11:25:18 -0700 (PDT)
Received: from smtp.mozilla.org (zlb1.mail.corp.phx1.mozilla.com [10.20.77.200])
	by zmmta1.mail.corp.phx1.mozilla.com (Postfix) with ESMTP id CE72610634D
	for <[e-mail]>; Wed, 11 Sep 2013 11:25:18 -0700 (PDT)
X-Virus-Scanned: amavisd-new at mozilla.org
Received: from buildbot-master81.srv.releng.scl3.mozilla.com (v-1030.fw1.releng.scl3.mozilla.net [63.245.214.82])
	by mx1.mail.corp.phx1.mozilla.com (Postfix) with SMTP id A7113F269C
	for <[e-mail]>; Wed, 11 Sep 2013 11:25:18 -0700 (PDT)
Content-Transfer-Encoding: base64
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Date: Wed, 11 Sep 2013 11:25:18 -0700
Subject: Try submission 6d8c5fb4f486
From: [e-mail]
References: <try-6d8c5fb4f486>
In-Reply-To: <try-6d8c5fb4f486>
CC: 
To: [e-mail]
Message-Id: <[e-mail]>

Return-Path: [e-mail]
Received: from zmmta1.mail.corp.phx1.mozilla.com (LHLO
 zmmta1.mail.corp.phx1.mozilla.com) (10.20.77.21) by
 zmmbox6.mail.corp.phx1.mozilla.com with LMTP; Wed, 11 Sep 2013 11:25:09
 -0700 (PDT)
Received: from zmmta1.mail.corp.phx1.mozilla.com (localhost6.localdomain [127.0.0.1])
	by zmmta1.mail.corp.phx1.mozilla.com (Postfix) with ESMTP id EC5701063E8
	for <[e-mail]>; Wed, 11 Sep 2013 11:25:08 -0700 (PDT)
Received: from smtp.mozilla.org (zlb1.mail.corp.phx1.mozilla.com [10.20.77.200])
	by zmmta1.mail.corp.phx1.mozilla.com (Postfix) with ESMTP id E73181063E6
	for <[e-mail]>; Wed, 11 Sep 2013 11:25:08 -0700 (PDT)
X-Virus-Scanned: amavisd-new at mozilla.org
Received: from localhost6.localdomain6 (vpn1.releng.scl3.mozilla.com [63.245.214.124])
	by mx1.mail.corp.phx1.mozilla.com (Postfix) with SMTP id AA235F2728
	for <[e-mail]>; Wed, 11 Sep 2013 11:25:08 -0700 (PDT)
Content-Transfer-Encoding: base64
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Date: Wed, 11 Sep 2013 11:25:08 -0700
Subject: Try submission 6d8c5fb4f486
From: [e-mail]
References: <try-6d8c5fb4f486>
In-Reply-To: <try-6d8c5fb4f486>
CC: 
To: [e-mail]
Message-Id: <[e-mail]>

Can provide more headers upon request.

Comment 8

[Nathan Froyd [:froydnj]]

Seeing this on try pushes this morning still.

Comment 9

[Chris AtLee [:catlee]]

Very strange.

I see in the logs the email being sent at 11:25:18. I don't see the one from 11:25:08. Also, the one from 11:25:08 looks like it was sent by a VPN user.

Comment 10

[Chris AtLee [:catlee]]

Looks like these emails aren't being generated by releng. I think we have a good lead on where they're coming from, and are reaching out to them now.

Comment 11

[Chris AtLee [:catlee]]

should be good now.