Closed Bug 803680 Opened 10 years ago Closed 10 years ago

multimaster releases broke preproduction

Categories

(Release Engineering :: Release Automation: Other, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bhearsum, Assigned: bhearsum)

Details

Attachments

(4 files, 1 obsolete file)

change(s) NOT sent, something went wrong:
[Failure instance: Traceback from remote host -- Traceback (most recent call last):
  File "/builds/buildbot/release-master/lib/python2.6/site-packages/twisted/spread/pb.py", line 1346, in remote_respond
    d = self.portal.login(self, mind, IPerspective)
  File "/builds/buildbot/release-master/lib/python2.6/site-packages/twisted/cred/portal.py", line 116, in login
    ).addCallback(self.realm.requestAvatar, mind, *interfaces
  File "/builds/buildbot/release-master/lib/python2.6/site-packages/twisted/internet/defer.py", line 260, in addCallback
    callbackKeywords=kw)
  File "/builds/buildbot/release-master/lib/python2.6/site-packages/twisted/internet/defer.py", line 249, in addCallbacks
    self._runCallbacks()
--- <exception caught here> ---
  File "/builds/buildbot/release-master/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
    self.result = callback(self.result, *args, **kw)
  File "/builds/buildbot/release-master/lib/python2.6/site-packages/buildbot-0.8.2_hg_bd4812420e63_production_0.8-py2.6.egg/buildbot/master.py", line 481, in requestAvatar
    raise ValueError("no PBChangeSource installed")
exceptions.ValueError: no PBChangeSource installed
]

http://preproduction-master.srv.releng.scl3.mozilla.com:8710/builders/release-master/builds/215
In an ideal world we'd use a separate builder + scheduler, but preproduction isn't that far along...
Attachment #673971 - Flags: review?(rail)
Comment on attachment 673971 [details] [diff] [review]
use universal master for preprod releases

ship it
Attachment #673971 - Flags: review?(rail) → review+
Comment on attachment 673971 [details] [diff] [review]
use universal master for preprod releases

Review of attachment 673971 [details] [diff] [review]:
-----------------------------------------------------------------

Rebuilding in http://preproduction-master.srv.releng.scl3.mozilla.com:8710/builders/release-master/builds/217 to test this.
Attachment #673971 - Flags: checked-in+
Attachment #673990 - Flags: review?(rail) → review+
Attachment #674006 - Flags: review?(rail)
Attachment #674006 - Flags: review?(rail) → review+
Attachment #673992 - Attachment is obsolete: true
Attachment #673992 - Flags: review?(rail)
Attachment #674006 - Flags: checked-in+
Attachment #673990 - Flags: checked-in+
I triggered another job. Because release master recreates itself I don't think I needed to reconfig before doing so...
Still didn't work.
I just noticed that UNIVERSAL=1 still isn't in the log for the release-master: http://preproduction-master.srv.releng.scl3.mozilla.com:8710/builders/release-master/builds/220/steps/shell_7/logs/stdio

And then I found a pyc file that was many months old (release_master.pyc). I had to restart the master to get it up-to-date, now I've got a new problem:
/builds/buildbot/release-master/lib/python2.6/site-packages/twisted/mail/smtp.py:10: DeprecationWarning: the MimeWriter module is deprecated; use the email package instead
  import MimeWriter, tempfile, rfc822
Traceback (most recent call last):
  File "/builds/buildbot/release-master/lib/python2.6/site-packages/buildbot-0.8.2_hg_bd4812420e63_production_0.8-py2.6.egg/buildbot/scripts/runner.py", line 1042, in doCheckConfig
    ConfigLoader(configFileName=configFileName)
  File "/builds/buildbot/release-master/lib/python2.6/site-packages/buildbot-0.8.2_hg_bd4812420e63_production_0.8-py2.6.egg/buildbot/scripts/checkconfig.py", line 31, in __init__
    self.loadConfig(configFile, check_synchronously_only=True)
  File "/builds/buildbot/release-master/lib/python2.6/site-packages/buildbot-0.8.2_hg_bd4812420e63_production_0.8-py2.6.egg/buildbot/master.py", line 833, in loadConfig
    "%s uses unknown builder %s" % (s, b)
AssertionError: <buildbotcustom.scheduler.AggregatingScheduler instance at 0x170c29e0> uses unknown builder release-mozilla-beta-start_uptake_monitoring
make[1]: *** [checkconfig] Error 1
make[1]: Leaving directory `/builds/buildbot/release-master

"yay"
I tried this local change on the preproduction release-master and it got it to the point of passing checkconfig. Turns out we don't create the builder if disableBouncerEntries is True, but still create the Scheduler.
Attachment #676242 - Flags: review?(rail)
Attachment #676242 - Flags: review?(rail) → review+
Comment on attachment 676242 [details] [diff] [review]
don't create start uptake monitoring scheduler when bouncer entries are disabled

Retriggered another preprod release run.
Attachment #676242 - Flags: checked-in+
I think this is working again. This job passed, at least: http://preproduction-master.srv.releng.scl3.mozilla.com:8710/builders/release-master/builds/223
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.