release-runner isn't sending mail on release-sanity failures

RESOLVED FIXED

Status

Release Engineering
Release Automation: Other
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: nthomas, Assigned: nthomas)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Assignee)

Description

3 years ago
This is the second time I've seen this in the last few days:

2015-03-30 13:59:52,699 - INFO - command: ERROR
Traceback (most recent call last):
  File "/builds/releaserunner/tools/lib/python/util/commands.py", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['python', 'tools/buildbot-helpers/release_sanity.py', '--branch', 'mozilla-beta', '--username', 'sledru@mozilla.com', '--version', '38.0b1', '--build-number', '1', '--release-config', 'release-fennec-mozilla-beta.py', '--skip-verify-configs', '--masters-json-file', 'https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-masters.json', '--configs-dir', 'buildbot-configs', '--configs-branch', 'production', '--dry-run']' returned non-zero exit status 1
2015-03-30 13:59:52,718 - INFO - command: END (32.64s elapsed)

2015-03-30 13:59:52,718 - INFO - mark as failed Fennec-38.0b1-build1
2015-03-30 13:59:52,719 - DEBUG - Request to https://ship-it.mozilla.org/releases/Fennec-38.0b1-build1
2015-03-30 13:59:52,720 - INFO - Resetting dropped connection: ship-it.mozilla.org
2015-03-30 13:59:52,853 - DEBUG - "POST /releases/Fennec-38.0b1-build1 HTTP/1.1" 200 0
2015-03-30 13:59:52,854 - INFO - mark as failed Firefox-38.0b1-build1
2015-03-30 13:59:52,854 - DEBUG - Request to https://ship-it.mozilla.org/releases/Firefox-38.0b1-build1
2015-03-30 13:59:52,903 - DEBUG - "POST /releases/Firefox-38.0b1-build1 HTTP/1.1" 200 0
2015-03-30 13:59:52,904 - DEBUG - Releasing lock: /builds/releaserunner/tools/buildfarm/release/.release-runner.lock
Traceback (most recent call last):
  File "release-runner.py", line 482, in <module>
    main(options)
  File "release-runner.py", line 404, in main
    ssh_username=hg_username, ssh_key=hg_ssh_key)
  File "/builds/releaserunner/tools/lib/python/util/hg.py", line 593, in apply_and_push
    changer(localrepo, 1)
  File "release-runner.py", line 393, in process_configs
    ['--dry-run'])
  File "/builds/releaserunner/tools/lib/python/util/commands.py", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['python', 'tools/buildbot-helpers/release_sanity.py', '--branch', 'mozilla-beta', '--username', 'sledru@mozilla.com', '--version', '38.0b1', '--build-number', '1', '--release-config', 'release-fennec-mozilla-beta.py', '--skip-verify-configs', '--masters-json-file', 'https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-masters.json', '--configs-dir', 'buildbot-configs', '--configs-branch', 'production', '--dry-run']' returned non-zero exit status 1

There's no indication of failure unless you look at https://ship-it.mozilla.org/releases.html, or wonder why no release jobs are running. Perhaps something to do with changing where the mail goes to.
I thought I fixed this bug 1146336 already...
Assignee: nobody → bhearsum
AFAICT the e-mail address is being extracted correctly now:
grep "notify_to:" release-runner.ini|perl -pe 's/.*?<(.*?)>/$1 /g'
release-automation-notifications@mozilla.com 

And it appears that mail was at least attempted to be sent:
Mar 30 13:59:56 buildbot-master81.bb.releng.scl3.mozilla.com postfix/smtp[27804]: 3C8D517FD38: to=<release-automation-notifications@mozilla.com>, relay=smtp.mail.scl3.mozilla.com[10.22.77.61]:25, delay=0.04, delays=0.03/0/0.01/0.01, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 46269C1A2B)

Perhaps it got hung up or marked as spam?
(In reply to Ben Hearsum [:bhearsum] from comment #2)
> AFAICT the e-mail address is being extracted correctly now:
> grep "notify_to:" release-runner.ini|perl -pe 's/.*?<(.*?)>/$1 /g'
> release-automation-notifications@mozilla.com 
> 
> And it appears that mail was at least attempted to be sent:
> Mar 30 13:59:56 buildbot-master81.bb.releng.scl3.mozilla.com
> postfix/smtp[27804]: 3C8D517FD38:
> to=<release-automation-notifications@mozilla.com>,
> relay=smtp.mail.scl3.mozilla.com[10.22.77.61]:25, delay=0.04,
> delays=0.03/0/0.01/0.01, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as
> 46269C1A2B)
> 
> Perhaps it got hung up or marked as spam?

Lawrence, do you have access enough to the release-automation-notifications list to see if there's a "[release-runner] failed" message that got marked as spam or otherwise hung up?
Flags: needinfo?(lmandel)
Unfortunately, I don't see any way to view discarded or flagged messages in the Google interface.
Flags: needinfo?(lmandel)
Lately, we had this issue at multiple occasions (especially with the build of beta releases from m-r).
This bug caused some pain.
Flags: needinfo?(catlee)
(Assignee)

Updated

3 years ago
Duplicate of this bug: 1158130
Sorry, should've unassigned myself from this after my initial investigation dead ended. The other work on my plate right now is higher priority than this AFAIK, so I won't have time to look at this in the next couple of weeks unless something changes.
Assignee: bhearsum → nobody
(Assignee)

Comment 8

3 years ago
I compared the messages which are getting through, like the 'Build of Firefox-38.0.5b1-build1' and the slow reconfig warning, with the release runner failures. The working messages use a from of 'release@mozilla.com', while the rr failures use 'cltbld@buildbot-master81.bb.releng.scl3.mozilla.com' because we don't specify anything else. Something must think that's a bad sender.
Assignee: nobody → nthomas
Flags: needinfo?(catlee)
OS: Mac OS X → All
(Assignee)

Comment 9

3 years ago
Created attachment 8604129 [details] [diff] [review]
[tools] Specify 'from' when calling mail
Attachment #8604129 - Flags: review?(bhearsum)
Comment on attachment 8604129 [details] [diff] [review]
[tools] Specify 'from' when calling mail

Review of attachment 8604129 [details] [diff] [review]:
-----------------------------------------------------------------

Nice sleuthing!
Attachment #8604129 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 11

3 years ago
Comment on attachment 8604129 [details] [diff] [review]
[tools] Specify 'from' when calling mail

With a s/NOTIFY_TO/NOTIFY_FROM/ in the new block of code:
https://hg.mozilla.org/build/tools/rev/de9b30d320f0

Deployed to production, was rev 2fa0b22073d2.
Attachment #8604129 - Flags: checked-in+
(Assignee)

Comment 12

3 years ago
I'm going to close this, we can reopen if there are further issues.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.