Closed
Bug 1149316
Opened 9 years ago
Closed 9 years ago
release-runner isn't sending mail on release-sanity failures
Categories
(Release Engineering :: Release Automation: Other, defect)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: nthomas, Assigned: nthomas)
References
Details
Attachments
(1 file)
1.21 KB,
patch
|
bhearsum
:
review+
nthomas
:
checked-in+
|
Details | Diff | Splinter Review |
This is the second time I've seen this in the last few days: 2015-03-30 13:59:52,699 - INFO - command: ERROR Traceback (most recent call last): File "/builds/releaserunner/tools/lib/python/util/commands.py", line 51, in run_cmd return subprocess.check_call(cmd, **kwargs) File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '['python', 'tools/buildbot-helpers/release_sanity.py', '--branch', 'mozilla-beta', '--username', 'sledru@mozilla.com', '--version', '38.0b1', '--build-number', '1', '--release-config', 'release-fennec-mozilla-beta.py', '--skip-verify-configs', '--masters-json-file', 'https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-masters.json', '--configs-dir', 'buildbot-configs', '--configs-branch', 'production', '--dry-run']' returned non-zero exit status 1 2015-03-30 13:59:52,718 - INFO - command: END (32.64s elapsed) 2015-03-30 13:59:52,718 - INFO - mark as failed Fennec-38.0b1-build1 2015-03-30 13:59:52,719 - DEBUG - Request to https://ship-it.mozilla.org/releases/Fennec-38.0b1-build1 2015-03-30 13:59:52,720 - INFO - Resetting dropped connection: ship-it.mozilla.org 2015-03-30 13:59:52,853 - DEBUG - "POST /releases/Fennec-38.0b1-build1 HTTP/1.1" 200 0 2015-03-30 13:59:52,854 - INFO - mark as failed Firefox-38.0b1-build1 2015-03-30 13:59:52,854 - DEBUG - Request to https://ship-it.mozilla.org/releases/Firefox-38.0b1-build1 2015-03-30 13:59:52,903 - DEBUG - "POST /releases/Firefox-38.0b1-build1 HTTP/1.1" 200 0 2015-03-30 13:59:52,904 - DEBUG - Releasing lock: /builds/releaserunner/tools/buildfarm/release/.release-runner.lock Traceback (most recent call last): File "release-runner.py", line 482, in <module> main(options) File "release-runner.py", line 404, in main ssh_username=hg_username, ssh_key=hg_ssh_key) File "/builds/releaserunner/tools/lib/python/util/hg.py", line 593, in apply_and_push changer(localrepo, 1) File "release-runner.py", line 393, in process_configs ['--dry-run']) File "/builds/releaserunner/tools/lib/python/util/commands.py", line 51, in run_cmd return subprocess.check_call(cmd, **kwargs) File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['python', 'tools/buildbot-helpers/release_sanity.py', '--branch', 'mozilla-beta', '--username', 'sledru@mozilla.com', '--version', '38.0b1', '--build-number', '1', '--release-config', 'release-fennec-mozilla-beta.py', '--skip-verify-configs', '--masters-json-file', 'https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-masters.json', '--configs-dir', 'buildbot-configs', '--configs-branch', 'production', '--dry-run']' returned non-zero exit status 1 There's no indication of failure unless you look at https://ship-it.mozilla.org/releases.html, or wonder why no release jobs are running. Perhaps something to do with changing where the mail goes to.
Comment 2•9 years ago
|
||
AFAICT the e-mail address is being extracted correctly now: grep "notify_to:" release-runner.ini|perl -pe 's/.*?<(.*?)>/$1 /g' release-automation-notifications@mozilla.com And it appears that mail was at least attempted to be sent: Mar 30 13:59:56 buildbot-master81.bb.releng.scl3.mozilla.com postfix/smtp[27804]: 3C8D517FD38: to=<release-automation-notifications@mozilla.com>, relay=smtp.mail.scl3.mozilla.com[10.22.77.61]:25, delay=0.04, delays=0.03/0/0.01/0.01, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 46269C1A2B) Perhaps it got hung up or marked as spam?
Comment 3•9 years ago
|
||
(In reply to Ben Hearsum [:bhearsum] from comment #2) > AFAICT the e-mail address is being extracted correctly now: > grep "notify_to:" release-runner.ini|perl -pe 's/.*?<(.*?)>/$1 /g' > release-automation-notifications@mozilla.com > > And it appears that mail was at least attempted to be sent: > Mar 30 13:59:56 buildbot-master81.bb.releng.scl3.mozilla.com > postfix/smtp[27804]: 3C8D517FD38: > to=<release-automation-notifications@mozilla.com>, > relay=smtp.mail.scl3.mozilla.com[10.22.77.61]:25, delay=0.04, > delays=0.03/0/0.01/0.01, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as > 46269C1A2B) > > Perhaps it got hung up or marked as spam? Lawrence, do you have access enough to the release-automation-notifications list to see if there's a "[release-runner] failed" message that got marked as spam or otherwise hung up?
Flags: needinfo?(lmandel)
Comment 4•9 years ago
|
||
Unfortunately, I don't see any way to view discarded or flagged messages in the Google interface.
Flags: needinfo?(lmandel)
Comment 5•9 years ago
|
||
Lately, we had this issue at multiple occasions (especially with the build of beta releases from m-r). This bug caused some pain.
Updated•9 years ago
|
Flags: needinfo?(catlee)
Comment 7•9 years ago
|
||
Sorry, should've unassigned myself from this after my initial investigation dead ended. The other work on my plate right now is higher priority than this AFAIK, so I won't have time to look at this in the next couple of weeks unless something changes.
Assignee: bhearsum → nobody
Assignee | ||
Comment 8•9 years ago
|
||
I compared the messages which are getting through, like the 'Build of Firefox-38.0.5b1-build1' and the slow reconfig warning, with the release runner failures. The working messages use a from of 'release@mozilla.com', while the rr failures use 'cltbld@buildbot-master81.bb.releng.scl3.mozilla.com' because we don't specify anything else. Something must think that's a bad sender.
Assignee: nobody → nthomas
Flags: needinfo?(catlee)
OS: Mac OS X → All
Assignee | ||
Comment 9•9 years ago
|
||
Attachment #8604129 -
Flags: review?(bhearsum)
Comment 10•9 years ago
|
||
Comment on attachment 8604129 [details] [diff] [review] [tools] Specify 'from' when calling mail Review of attachment 8604129 [details] [diff] [review]: ----------------------------------------------------------------- Nice sleuthing!
Attachment #8604129 -
Flags: review?(bhearsum) → review+
Assignee | ||
Comment 11•9 years ago
|
||
Comment on attachment 8604129 [details] [diff] [review] [tools] Specify 'from' when calling mail With a s/NOTIFY_TO/NOTIFY_FROM/ in the new block of code: https://hg.mozilla.org/build/tools/rev/de9b30d320f0 Deployed to production, was rev 2fa0b22073d2.
Attachment #8604129 -
Flags: checked-in+
Assignee | ||
Comment 12•9 years ago
|
||
I'm going to close this, we can reopen if there are further issues.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•