755067 - build_scheduler errors on 'assert state_json is not None'

Status: RESOLVED WORKSFORME

(Release Engineering :: Release Automation: Other, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Since this morning we've been hitting this on bm36's build_scheduler:

2012-05-14 11:01:10-0700 [-] Unhandled Error
        Traceback (most recent call last):
          File "/usr/lib64/python2.6/threading.py", line 504, in __bootstrap
            self.__bootstrap_inner()
          File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
            self.run()          File "/usr/lib64/python2.6/threading.py", line 484, in run
            self.__target(*self.__args, **self.__kwargs)
        --- <exception caught here> ---
          File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/twisted/python/threadpool.py", line 207, in _worker
            result = context.call(ctx, function, *args, **kwargs)          File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/twisted/python/context.py", line 59, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/twisted/python/context.py", line 37, in callWithContext
            return func(*args,**kw)
          File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/twisted/enterprise/adbapi.py", line 429, in _runInteraction
            result = interaction(trans, *args, **kw)
          File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/schedulers/timed.py", line 238, in _check_timer
            s = self.get_state(t)
          File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/schedulers/base.py", line 90, in get_state            return self.parent.db.scheduler_get_state(self.schedulerid, t)
          File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/db/connector.py", line 656, in scheduler_g
et_state            assert state_json is not None
        exceptions.AssertionError: 

Reconfig's happened at :
  2012-05-14 12:27:08-0700 [-] configuration update complete
  2012-05-14 12:27:08-0700 [-] configuration update complete
  2012-05-14 14:59:24-0700 [-] configuration update complete
The double reconfig at 12:27 is a little odd, but otherwise this isn't a smoking gun.

I checked the scheduler db and none of the rows in scheduler table have a state field of length 0. Restarting build_scheduler resolved the problem. Fallout from bug 754885 ?

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Or the db going ro ?

Comment 2

[Chris AtLee [:catlee]]

Fallout from bug 754885 - I deleted all the pine schedulers since they had a huge backlog of changes to go through. The test schedulers all got recreated when I restarted the tests scheduler master, but I neglected to restart the build scheduler master,so it still had a reference to the old (deleted) pine build scheduler.