1149316 - release-runner isn't sending mail on release-sanity failures

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

This is the second time I've seen this in the last few days:

2015-03-30 13:59:52,699 - INFO - command: ERROR
Traceback (most recent call last):
  File "/builds/releaserunner/tools/lib/python/util/commands.py", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['python', 'tools/buildbot-helpers/release_sanity.py', '--branch', 'mozilla-beta', '--username', '[e-mail]', '--version', '38.0b1', '--build-number', '1', '--release-config', 'release-fennec-mozilla-beta.py', '--skip-verify-configs', '--masters-json-file', 'https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-masters.json', '--configs-dir', 'buildbot-configs', '--configs-branch', 'production', '--dry-run']' returned non-zero exit status 1
2015-03-30 13:59:52,718 - INFO - command: END (32.64s elapsed)

2015-03-30 13:59:52,718 - INFO - mark as failed Fennec-38.0b1-build1
2015-03-30 13:59:52,719 - DEBUG - Request to https://ship-it.mozilla.org/releases/Fennec-38.0b1-build1
2015-03-30 13:59:52,720 - INFO - Resetting dropped connection: ship-it.mozilla.org
2015-03-30 13:59:52,853 - DEBUG - "POST /releases/Fennec-38.0b1-build1 HTTP/1.1" 200 0
2015-03-30 13:59:52,854 - INFO - mark as failed Firefox-38.0b1-build1
2015-03-30 13:59:52,854 - DEBUG - Request to https://ship-it.mozilla.org/releases/Firefox-38.0b1-build1
2015-03-30 13:59:52,903 - DEBUG - "POST /releases/Firefox-38.0b1-build1 HTTP/1.1" 200 0
2015-03-30 13:59:52,904 - DEBUG - Releasing lock: /builds/releaserunner/tools/buildfarm/release/.release-runner.lock
Traceback (most recent call last):
  File "release-runner.py", line 482, in <module>
    main(options)
  File "release-runner.py", line 404, in main
    ssh_username=hg_username, ssh_key=hg_ssh_key)
  File "/builds/releaserunner/tools/lib/python/util/hg.py", line 593, in apply_and_push
    changer(localrepo, 1)
  File "release-runner.py", line 393, in process_configs
    ['--dry-run'])
  File "/builds/releaserunner/tools/lib/python/util/commands.py", line 51, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['python', 'tools/buildbot-helpers/release_sanity.py', '--branch', 'mozilla-beta', '--username', '[e-mail]', '--version', '38.0b1', '--build-number', '1', '--release-config', 'release-fennec-mozilla-beta.py', '--skip-verify-configs', '--masters-json-file', 'https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-masters.json', '--configs-dir', 'buildbot-configs', '--configs-branch', 'production', '--dry-run']' returned non-zero exit status 1

There's no indication of failure unless you look at https://ship-it.mozilla.org/releases.html, or wonder why no release jobs are running. Perhaps something to do with changing where the mail goes to.

Comment 1

[[e-mail] (:bhearsum)]

I thought I fixed this bug 1146336 already...

Comment 2

[[e-mail] (:bhearsum)]

AFAICT the e-mail address is being extracted correctly now:
grep "notify_to:" release-runner.ini|perl -pe 's/.*?<(.*?)>/$1 /g'
[e-mail] 

And it appears that mail was at least attempted to be sent:
Mar 30 13:59:56 buildbot-master81.bb.releng.scl3.mozilla.com postfix/smtp[27804]: 3C8D517FD38: to=<[e-mail]>, relay=smtp.mail.scl3.mozilla.com[10.22.77.61]:25, delay=0.04, delays=0.03/0/0.01/0.01, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 46269C1A2B)

Perhaps it got hung up or marked as spam?

Comment 3

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #2)
> AFAICT the e-mail address is being extracted correctly now:
> grep "notify_to:" release-runner.ini|perl -pe 's/.*?<(.*?)>/$1 /g'
> [e-mail] 
> 
> And it appears that mail was at least attempted to be sent:
> Mar 30 13:59:56 buildbot-master81.bb.releng.scl3.mozilla.com
> postfix/smtp[27804]: 3C8D517FD38:
> to=<[e-mail]>,
> relay=smtp.mail.scl3.mozilla.com[10.22.77.61]:25, delay=0.04,
> delays=0.03/0/0.01/0.01, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as
> 46269C1A2B)
> 
> Perhaps it got hung up or marked as spam?

Lawrence, do you have access enough to the release-automation-notifications list to see if there's a "[release-runner] failed" message that got marked as spam or otherwise hung up?

Comment 4

[Lawrence Mandel [:lmandel] (use needinfo)]

Unfortunately, I don't see any way to view discarded or flagged messages in the Google interface.

Comment 5

[Sylvestre Ledru [:Sylvestre]]

Lately, we had this issue at multiple occasions (especially with the build of beta releases from m-r).
This bug caused some pain.

Comment 7

[[e-mail] (:bhearsum)]

Sorry, should've unassigned myself from this after my initial investigation dead ended. The other work on my plate right now is higher priority than this AFAIK, so I won't have time to look at this in the next couple of weeks unless something changes.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

I compared the messages which are getting through, like the 'Build of Firefox-38.0.5b1-build1' and the slow reconfig warning, with the release runner failures. The working messages use a from of '[e-mail]', while the rr failures use '[e-mail]' because we don't specify anything else. Something must think that's a bad sender.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] Specify 'from' when calling mail

Comment 10

[[e-mail] (:bhearsum)]

Comment on attachment 8604129 [details] [diff] [review]
[tools] Specify 'from' when calling mail

Review of attachment 8604129 [details] [diff] [review]:
-----------------------------------------------------------------

Nice sleuthing!

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8604129 [details] [diff] [review]
[tools] Specify 'from' when calling mail

With a s/NOTIFY_TO/NOTIFY_FROM/ in the new block of code:
https://hg.mozilla.org/build/tools/rev/de9b30d320f0

Deployed to production, was rev 2fa0b22073d2.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

I'm going to close this, we can reopen if there are further issues.