628451 - mozilla-central push ac6c96109dcf did not show up in pending builds or buildbot web interface

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

This push is in the pushlog but isn't showing up in the buildbot webui.

I have followed the links in the db and found that all of the builds were claimed by pm03 with the same pid/time and have complete=1, results=NULL and submitted_at of 1295910189 with a complete_at time of either 1295910191 or 1295910192.  

The mysql query output is too large to paste here but is at: http://pastebin.mozilla.org/979633 .

Comment 1

[John Ford [:jhford] CET/CEST Berlin Time]

not sure if related, but there are a lot of exceptions on this master that look like:

    2011-01-24 16:14:59-0800 [Broker,4142,10.250.50.207] <Build WINNT 5.2 tracemonkey build>.startBuild
    2011-01-24 16:14:59-0800 [Broker,4142,10.250.50.207] error while creating step, factory=buildbotcustom.steps.source.MercurialCloneCommand, args={'workdir': '.', 'command': ['hg', 'clone', 'http://hg.mozilla.org/build/tools', 'tools'], 'name': 'clone_buildtools', 'description': ['clone', 'build tools']}
    2011-01-24 16:14:59-0800 [Broker,4142,10.250.50.207] Build.setupBuild failed
    2011-01-24 16:14:59-0800 [Broker,4142,10.250.50.207] Unhandled Error
            Traceback (most recent call last):
              File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/twisted/internet/defer.py", line 318, in callback
                self._startRunCallbacks(result)
              File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/twisted/internet/defer.py", line 424, in _startRunCallbacks
                self._runCallbacks()
              File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
                self.result = callback(self.result, *args, **kw)
              File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/builder.py", line 896, in _startBuild_2
                d = build.startBuild(bs, self.expectations, sb)
            --- <exception caught here> ---
              File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/base.py", line 217, in startBuild
                self.setupBuild(expectations) # create .steps
              File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/base.py", line 272, in setupBuild
                step = factory(**args)
              File "/tools/buildbotcustom/buildbotcustom/steps/source.py", line 12, in __init__
                self.super_class.__init__(self, log_eval_func=log_eval_func, **kwargs)
            exceptions.TypeError: unbound method __init__() must be called with C instance as first argument (got MercurialCloneCommand instance instead)

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

It looks like an unsuccessful reconfig will produce this error.  The dynamically-created classes have a different inheritance hierarchy, even if their names are the same, and as a result __init__ gets upset.

I did an unsuccessful reconfig on this master, but trusted the buildmaster when it said "will continue using the old config instead".

I've now run a successful reconfig, and things are looking much better.

We should really fix this reconfig problem.  I was definitely at fault for doing an incorrect reconfig like that, but still -- it shouldn't kill the master.

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

I have verified that jobs are able to start on this master without hitting this issue.